From bbe73308a73c08bf4fa789e44a825541fff08149 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Wed, 3 Apr 2024 17:03:20 -0700 Subject: [PATCH 01/67] Adding the CAPI integrations dropdown and overview page to Rancher latest, v2.8, and v2.7. Adjusting the sidebars for the respective versions as well. Signed-off-by: Sunil Singh --- .../cluster-api/capi.md | 14 ++ .../cluster-api/overview.md | 210 ++++++++++++++++++ sidebars.js | 11 + static/img/30000ft_view.png | Bin 0 -> 103225 bytes .../cluster-api/capi.md | 14 ++ .../cluster-api/overview.md | 210 ++++++++++++++++++ .../cluster-api/capi.md | 14 ++ .../cluster-api/overview.md | 210 ++++++++++++++++++ versioned_sidebars/version-2.7-sidebars.json | 11 + versioned_sidebars/version-2.8-sidebars.json | 11 + 10 files changed, 705 insertions(+) create mode 100644 docs/integrations-in-rancher/cluster-api/capi.md create mode 100644 docs/integrations-in-rancher/cluster-api/overview.md create mode 100644 static/img/30000ft_view.png create mode 100644 versioned_docs/version-2.7/integrations-in-rancher/cluster-api/capi.md create mode 100644 versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md create mode 100644 versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md create mode 100644 versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md diff --git a/docs/integrations-in-rancher/cluster-api/capi.md b/docs/integrations-in-rancher/cluster-api/capi.md new file mode 100644 index 00000000000..53ec6f4a66f --- /dev/null +++ b/docs/integrations-in-rancher/cluster-api/capi.md @@ -0,0 +1,14 @@ +--- +title: Cluster API with Rancher Turtles +--- + + + + + +Rancher Turtles is a [Rancher extension](../rancher-extensions.md) that provides integration between your Cluster API (CAPI) and Rancher to manage the lifecycle of provisioned Kubernetes clusters. The main functionality is outlined below: + +- Importing CAPI clusters into Rancher: Installing the Rancher Cluster Agent in CAPI provisioned clusters. +- Configuration support for the CAPI Operator. + +The [Overview](./overview.md) section outlines installation options, the main architecture, and a brief demo. For more detailed information, please view the [Rancher Turtles documentation](https://docs.rancher-turtles.com/docs/intro). diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md new file mode 100644 index 00000000000..4320dc87774 --- /dev/null +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -0,0 +1,210 @@ +--- +title: Overview +--- + + + + + +## Demo + + + +## Installing the Rancher Turtles Operator + +### Via Rancher Dashboard + +This is the recommended option for installing Rancher Turtles. + +Using the Rancher UI, and by adding the Turtles repository, we can let Rancher process the installation and configuration of the Cluster API Extension. + +:::caution +If you already have the Cluster API Operator installed in your cluster, you need to use the [manual helm installation method](./overview.md#via-helm-install) instead. +::: + +:::info prerequisite +Before installing Rancher Turtles in your Rancher environment, Rancher's `embedded-cluster-api` functionality must be disabled. This includes also cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. + +To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that applies these changes which are outlined below if manually removing: +- Disable the `embedded-cluster-api` feature in Rancher. +- Delete the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. +::: + +#### Installation + +- From your browser, access Rancher Manager and explore the **local** cluster. +- Using the left navigation panel, go to `Apps` -> `Repositories`. +- Click `Create` to add a new repository. +- Enter the following: + - **Name**: `turtles`. + - **Index URL**: https://rancher.github.io/turtles. +- Wait for the `turtles` repository to have a status of `Active`. +- Go to `Apps` -> `Charts`. +- Filter for `turtles`. +- Click `Rancher Turtles - the Cluster API Extension` +- Click `Install` -> `Next` -> `Install`. + +This will use the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or, alternatively, you can opt for the [manual installation via Helm](./overview.md#via-helm-install). If you are interested in learning more about the available values, you can check the Rancher Turtles [Helm chart reference guide](https://docs.rancher-turtles.com/docs/reference-guides/rancher-turtles-chart/values). + +The installation may take a few minutes and once completed you will see the following new deployments in the cluster: +- `rancher-turtles-system/rancher-turtles-controller-manager` +- `rancher-turtles-system/rancher-turtles-cluster-api-operator` +- `capi-system/capi-controller-manager` + +### Via Helm Install + +This section walks through different Helm installation options for Rancher Turtles depending on CAPI operator use: +- [Install Rancher Turtles with Cluster API Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency). +- [Install Rancher Turtles without Cluster API Operator](#install-rancher-turtles-without-cluster-api-operator-as-a-helm-dependency). + +The Cluster API Operator is required for installing Rancher Turtles and you can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. We recommend [installing as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) for the sake of simplicity, but the best option may depend on your specific configuration. + +CAPI Operator allows for handling the lifecycle of Cluster API providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). + +:::info prerequisite +Before installing Rancher Turtles in your Rancher environment, Rancher's `embedded-cluster-api` functionality must be disabled. This includes also cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. + +To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that applies these changes which are outlined below: +- Disable the `embedded-cluster-api` feature in Rancher. +- Delete the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. +::: + +#### Install Rancher Turtles with `Cluster API Operator` as a Helm dependency + +The `rancher-turtles` chart is available in https://rancher.github.io/turtles and this Helm repository must be added before proceeding with the installation: + +```bash +helm repo add turtles https://rancher.github.io/turtles +helm repo update +``` + +As mentioned before, installing Rancher Turtles requires the [Cluster API Operator](https://github.com/kubernetes-sigs/cluster-api-operator) and the Helm chart can handle its installation automatically with a minimum set of flags: + +```bash +helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ + -n rancher-turtles-system \ + --dependency-update \ + --create-namespace --wait \ + --timeout 180s +``` + +This operation could take a few minutes and once completed you can review the installed controllers listed below: +- `rancher-turtles-controller` +- `capi-operator` + +:::note +- If `cert-manager` is already available in the cluster, you can disable its installation as a Rancher Turtles dependency to avoid conflicts: +`--set cluster-api-operator.cert-manager.enabled=false` +- For a list of Rancher Turtles versions, refer to [Releases page](https://github.com/rancher/turtles/releases). +::: + +This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional Cluster API functionality. + +If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled. + +```bash +helm install ... + # Passing secret name and namespace for additional environment variables + --set cluster-api-operator.cluster-api.configSecret.name= +``` + +The following is an example of a user-managed secret `cluster-api-operator.cluster-api.configSecret.name=variables` with `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` feature flags set and an extra custom variable: + +```yaml title="secret.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: variables + namespace: rancher-turtles-system +type: Opaque +stringData: + CLUSTER_TOPOLOGY: "true" + EXP_CLUSTER_RESOURCE_SET: "true" + EXP_MACHINE_POOL: "true" + CUSTOM_ENV_VAR: "false" +``` + +:::info +For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://docs.rancher-turtles.com/docs/reference-guides/rancher-turtles-chart/values) +::: + +#### Install Rancher Turtles without `Cluster API Operator` as a Helm dependency + +:::note +Remember that if you opt for this installation option, you will need to manage the Cluster API Operator installation yourself. You can follow the [CAPI Operator guide](https://docs.rancher-turtles.com/docs/next/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. +::: + +The `rancher-turtles` chart is available in https://rancher.github.io/turtles and this Helm repository must be added before proceeding with the installation: + +```bash +helm repo add turtles https://rancher.github.io/turtles +helm repo update +``` + +and then it can be installed into the `rancher-turtles-system` namespace with: + +```bash +helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 + -n rancher-turtles-system + --set cluster-api-operator.enabled=false + --set cluster-api-operator.cluster-api.enabled=false + --create-namespace --wait + --dependency-update +``` + +As you can see, we are telling Helm to ignore installing `cluster-api-operator` as a dependency. + +This operation could take a few minutes and once completed you can review the installed controller listed below: +- `rancher-turtles-controller` + +## Uninstalling Rancher Turtles + +This section gives an overview of the Rancher Turtles uninstallation process. + +:::caution +When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the Cluster API Operator cleanup. This includes cleaning up Cluster API Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. + +To simplify uninstalling Rancher Turtles (via Rancher Manager or helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that applies these changes that are outlined below: +- Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. +- Deletes the CAPI `deployments` that are no longer needed. +::: + +To uninstall the Rancher Turtles Extension use the following helm command: + +```bash +helm uninstall -n rancher-turtles-system rancher-turtles --cascade foreground --wait +``` + +This may take a few minutes to complete. + +:::note +Remember that, if you use a different name for the installation or a different namespace, you may need to customize the command for your specific configuration. +::: + +Once uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: + +1. Create a `feature.yaml` file, with `embedded-cluster-api` set to true: +```yaml title="feature.yaml" +apiVersion: management.cattle.io/v3 +kind: Feature +metadata: + name: embedded-cluster-api +spec: + value: true +``` +2. Use `kubectl` to apply the `feature.yaml` file to the cluster: +```bash +kubectl apply -f feature.yaml +``` + +## Architecture Diagram + +Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the Cluster API (CAPI) for cluster management. + +![overview](/img/30000ft_view.png) + +## Security + +[SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. + +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, please visit the [Rancher Turtles Security](https://docs.rancher-turtles.com/docs/security/slsa) document. diff --git a/sidebars.js b/sidebars.js index 1ee600f77d4..ab01993973a 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1215,6 +1215,17 @@ const sidebars = { 'integrations-in-rancher/cloud-marketplace/supportconfig' ] }, + { + "type": "category", + "label": "Cluster API with Rancher Turtles", + "link": { + "type": "doc", + "id": "integrations-in-rancher/cluster-api/capi" + }, + "items": [ + "integrations-in-rancher/cluster-api/overview" + ] + }, { type: 'category', label: 'CIS Scans', diff --git a/static/img/30000ft_view.png b/static/img/30000ft_view.png new file mode 100644 index 0000000000000000000000000000000000000000..7441fa1b9cf270ddc549c5b1f1c5e11c7008fb45 GIT binary patch literal 103225 zcmeFZXIN8Pw>FGjaiiOUD2S+l2nZ;>V*?BbNbjK12^~TS&5labAfR-mw?ujg0W5%k zfYd-BfFd;%DIwCovE0wOp7UJi*ZcnX&X4b7m%v;rYtA|5827lx7;~;?w=`8*nAw?` zn3z~lH?He2G3_X4V%k}}cQ>@CnWf?3^UZ;dyklyR-x(M1QQeHA`{cp zZ6+q!*Gx=oZV8pz@^E4IU3Jy#Ok2o5DWuFOXxZm>!`PFF>3BTy-!>HH;VWp|_c zSIDZ?vMPA^DsUa-@{FMc5ZdSpKYx!A%`{8kXFH6BDdml4sn0g#Hzx62+l7(O+B1K< z{b@OH|K5J2!ENtOGvxE$wbR3YzI1Z?Ha4VT`0u?SGWZF(T)PYTl-~LOy+=IACGBUr zC-}Ic(qv_&LC?SdAGYI-Mh+ga-{&p>TSA0+)Abeh_fq*Re0Q^0dk1z}N$d z4G>+YWb&q+J&hqW=I+K<< zTZ4Wyd(vI-4roWpvqNs%q!WI2QPqBv$*bQ(!gHKhQ@Dkxn6D+FUt5#B^P5XD%$!^H zbiKtA2DFY%)N}0N-}Ld}mo3aB5&!y^8cOf<#FX^pZ(VUEt-V!y9{OseN4=&}+5Fe5 z5%m_**dsh%O?xfJ+;L_kH#7Nlis|m5FDm*l4>nNU6yu@;rFAxZ(StcP%;wibR+dIG zewg{|Y>r)coIG%ri(s(R9F#f~boZMRZ{}=|Sf?%fhvnC3w#Mu?gM-9U6&PJuiX3`O zJVST~6H@};cHw)rnN?L+#Rh+e1pU5~dcgOuTQ0e$TBLfD@TDpr_!wF0b+%OBk<%K% zv;2~&U@Ow9u+A+_Qzb*_o&>+9Z2SDb+R6pJ!)NL|2r_T+qt6`YwBOmCQ^+w58FZS% z!N~Ex$TuEEvFeNpVmb(d}Z0)q1>8ElzD_q$9T&r-pKK(=<5R%|w?s6u)QDZid{yVB| z;)TDZ4bc)k=YijQCb}j|cT|$3l!moe0~*}79H3aY|?Q+u~t-52Q>*$v47DtqL2 z|LxNDVITQ6=da@_NXv;mqBG_PU>?*<4?tG8L00X-zBe7l&_`R67V0vSSow-a2J9O{(*su?us6 z=`_!l)YPP2n+@dU+hne^PDsFfArUOr*5sD)J3$=^zgxyK2@7`4pG78Ak&i_N6WgB? z0@LMzR{vPACpTm8O=brE9n0B;`u6}vmWAuSze!Zk2h@W`5D6@DGe~NHKR`=LtH5Aoq4iu3|SO&RjYd3W& zy5e^qG3;38*E%h^ec&Zic7gLOck1X6)BkbUdQQ|N;c)Ef^mIGQpE0RNAVWs=C*~Xx zo|!gR3HAJ#p4j1?`R}Xn@c(qg=8xTVb%VUcz5!rlwe5hy z$XEZvm&a;=XDP{6bVP zbmP|d`TFt;!;(f2tF%S;xw`Ai?|n|dS7T3ZW2?0}!rFJuP=CA^W zaJKTlYr_Ju$&gbHqw}j#0a2M<$pfhoiL#Az0+#;k?zdBthGv(=k-k&w?TLBDRBq0N+JpGxcw>rDBlw>Q{i)3CkevQ zrpN0@M;*~!=^mlb;5w%U4-pnb4E?xwv&I75!{#XCusvGww55n43vu+^zlH)r>wOF| zvA_mBuZ$+ddd{HyV@EC~ptW*LSFog|^`oEiy*Un?9fOVqb~M|l6-b(`%EC0rKX z?k2vX9BooK|8$Jds^?))Kx(e|vCD8q=f;7h{j89>cAlIc zi`*1WDtM?;zZ-2dcY4=qWk2yG5S8nD%P${k7Rk|DO2afMSUAk-#6>5i3-X|dq&a|& zmctu9cODQt3>Oki_f~*y?FE3>c%a$ zluuHxJo`v*^POgN|EnJuKJGH)-aF4l?+NyRpGfzmX2TUPjXv4z()vN(J}u{PJin=G z`7V&@4nQP93f$%FypdAnc=Nds>IGV(Ii7jeL>J$S+v(T+y10b?@R2FZOa(#Ld4N^Q zWMJfTGfJ*yqn)F_D0Pt6T+&%91Nl?H!-0@E$Z!)p4fl)PW)Z$O_My>ujpXuj(wgtF z{(4$U4yox^m~Of!hrgjWs+^o~i@b0hI_bR$)OGkIIN%gxnH~YNn&M58rmSC%YvWze zFOR&EiV_KCJSDxFHx$#+s0)`Bz90LqZ}eq@&)l#mJ5L_ar8biV(maTw3V8A*vM^3p zY=dO(91eDPKJNz~>Ml2UqU7`EnnBGRbFyH6vUl-|ChwDfICh|6H-0 zw%%f_^4@v*3Lt201i0!rDLy+y@>Y2Bu$2CMZ<5CVcG9pc3+Ni&zh$4fdGxzweWKpz zPt<#mtNnHWgNP4@y-XkH0u=#u>n^h{P)ppvbO~KWTxsB*yMRa~5?r^4IpTigqXI)E zg1RpOjN|^^#&++M%y;%Y(3_?#lJ*roMz*~@CgNo$`I|PVmYFN+R4ol@T z%7#nRu?uFcu*e5blj!EahhmdY?*r1zldVD5F2T1(5mpRgU~0GpbD}`=hAb=5LL&>N zR{jX2Ve$yritJ!cR7BZS+D4`FtOVarV`m*&oE38pCD)~!<{Z)HGlSKf*&Qe5l*HxZ zK)6fJjUEk3Y%*WR_e#`)f<`oQ46vXtB&7I zO!-lwauxkmL7z!wIf*9O9gXoU@8G-BVGw&B(`?Q0%HePq@|-X8k$_fdt-lZ}VPg89 z5oj$t1XOA%k?~auQ=3!GiUz`-8elfS87?0Iqms7U!M)DP{7~w0P{%Dt8M+aP zVV;HKK;{b`Yn-?j-4HsqZY%e?K9 zehjIJKG3~ks>Qkf%K(MxbXV$kGZM5Y{~)bM)_ZHXRbn`Ym$&cKGbXhfIq5l`$nI&F9?b^La1B3ly>Mv`FNnlYQOCg<_^8dkPOc zuE=0cO+r?WruDMk-=U8$tYItQLWad_SI5*nn2R50-hFmbxG0B0`;46+A97_~R$X!? zPhjL_3UiZcYD^!F{~oux6uZO@l8-wK6l&MLgUbdWuX#R!F4cXycb!#^lue7Fa`qd> zkUce6pC=V-UWy|3Ke|M#bv(p5mRr#*c676seV0OUhqMLG<+&4DK4{~hLh#zx?9xoW z=}Q)C^DWX9vU1&MCHk!bSuLY{14d9jd@GEsdT{EteCYw-&LhiNBcF1;J{3KyNB5Lu zj>k~C6cz47?uHdW?Z?GeZGj3d^_1U8)#AeX$Ai{NIIY^Ym?yo3^Cvw(L*0m; zw}8nO)OY^K_)=<8F=J2iwiTx4b~HIg8yL`{<_?_bOy;V}PBn0m697}>mgloO#?PB; zgj-tZAgv0^SgJA$cD)Y1(0s8cxz()FP%C0uAX~Ut)8xh7&QhMh)r~t=fxlo0tUi|) zP%PI1hIolKxQYw|bt%sFC~l%aZ`-w@n>AN-6*ar>4|HI4qjm%iIH|P7@^R{eS|^^D zLeI3yr)v3oIaw_wE90}TiilACQFwsylx!eY;@LmHo@EJi*Cn+)RatV*%v8ZheC(RU zFSzK>m6+*1){u+mC|8J}-Db8R}SxnGOt+hw9r7Nh4%ntkJ#Oq5S; z{OHWl_GBw3Xmv!7da}=;{Z2KaraxuMZse>T*jx@KhWj5j-4Z?1+m_l+d&*4DW^)_b z%J@!)6e5%1S%9YfqOfR+8=LQWvLDuj6dcm_dhJRTt~a{mmu>?@!wT^xZe68hp40*iSFy%-JdR*)0XTvVHoLe4^_?MXOp)4mOV-bN_tRlm6btEKjxHDlW`>b-(=he6iB`g>sN~dds4E z!Nd3Tqf-+XEqAS!cXMb|=Wcb^ThYSv(kfq$-|3WU?(Rb=$dCxSia?%IGe#PpQeds*GEi0%fe0|>iO!#a^fUsuG z9_Npp1puut^~wxn?x;CE++aKQGyQk@-@G@oDXGl zX~7s#ls0591cF6;Gebxa>J=#Xcb_s2MxHr14z%}A{+e<%5byES=Lm!zp8+8s=Pr-o z{~F&`=u-N2(WESyF}YVmt2cFRcd-;DDd~OY2Z_@0ACp=-;S|a0Lz!TXQ{CC&E%gS0 znE}DtfsEQ|o{^I=CB;2~Rs{gp1#k30yo2m)*#N365stYCLORN z9LU+|>2hiHiw7qlmcwK{HyZ{CF_-K23g0_d1V$Tgy22xG&X!d@u#ns09^=NL;E>od z69ClT@|z!6SaQp1?}>Q@r|lDyonCFK;6yaStwydox%i1M)oHMi-snNo=E8T)c(Uff zve3gt7zWxjY#d}>KlA(oW4DDb#eQyatE}I*%4_6Myy+QZhBnyUyeqGHA!jleo-kV` zg<`U)yhHr{*VOc>^$B+WSa%dl*e}D0?8)-@PG*`0A$rOMC6>o zVF@%fM@96Y*UYNzRQ1-nsmsDJSu>vP!r!i&T&>-8@&qK^?~O(BH^FlJU!~mt$yBS? zd02MtvE}23X@mR!Gg+WIzJ2KDk^9>}@)zKEGKn>XK5Wpt^idT^;ua(H=*%V4BdT8Q zUwUt&bbQ~JA?hX}w%cqO3+|xX@1vu?4}_%muHD_qb~&^_p5Jiavp!tf9B#2>K3p1% z7}0HDU=r9{dsvSePHwm!WGB!kf8_FVMA}2d$jf6(t81sS4v%CAt=(M3HWPpe82}GD zRg+(I_9|tfh$W;1>!0*hqD1ZW;oNbh{`?S^*(@e5sTKhJl{)?9S)^oy+J#Uj!fo& znJj(6Z>K7bs6q=XfCyaIFnMtng>}%5+gvdUv#_W%#EApZL4HAf4y07;Tmpf-HsUj#_$~K`Cf-qi$n@c)f`u>IQp9BkGHhOXGilms|j|cj31`2 z6m57}5txu}>C?R`z+gsBf*U=`Ifj-mOK@d;KeAjMfJhVaJ>Tv+_1TGD%`a(jfRe%Q zJ`^quNG9hOA^j8r(LF4;C~DPbu`}YKGIOv~cr~`Vrut3L!`UF`P~eC^r>1$phwQ%o z8{M9bsfH((R~bEoQJX?~VYh0D{>N8FGXehZ>ksbF_Xz{13<{>by64jpg$wIwfAd99 zuYL*;0bqYU-y@%(TYsMyN;sU$FgW#|!3)Y+d?RdlyZxa-Q@_hFdiDSjP%*>EH3LvH zOxpV&W!i)9)D+wt^$c;RfVE(~oDh1ftYwm^keR{R(59&a77IKyHB(7bB12pWI2vR= zFyqqsLNHQs1xT)h^b}cfjI!xF>Y98vj#Suq4Wk(;y{bb!@1kZD*HuWK1wonr?Tf>Lla`N-@ zOMDL9B>`2AvcaTjX2r0T{u~e1Poi=yOTCfG(ST zPH5Y|%HLfhlTCy4$M@y7Va?$Gbb5^Cj5!n?Yn35HYqeneuz zqFw6Mz^U-{q|JcqL*DGizq`tpg&i|(b(Yz5y5HJQRfYjUs|dseBXltKdNlWHDf&Gp zZP6$+a=*4-R%H8=(cVLW{U_#(TO}@jv`pRO{q0Nd@NNRo>&*62p>c&+S~%d|x7Vi? zygXIot=Ok%oyLrr+P%Ty6aB3jgdZbUk4}u(>PC$kHT#DLM)m}9PSEh%3 zYiM})YqFEn=sj(}PXMiT4ieJ@mSXbib^c@ByU_{8Y1O*bv zxjppIV6DUsl;P5VXGZe~QGKriITI+1iqtvCiUx~0^*S-_(dTLGxWBuXPs`IegrFmg z;6VTtvlVYz!k>Q&u`vjGcu;=I(FnhUJDw*e@?^V+d%**sgZfK5k~H)jY)X!4KGO%s z`yuF5T+(}0)8#jRaBy3y}KC(#$5H$78mGxd=Btpu-4sl6?n)DgjROL}Mrq9_BFc zLX%xGu!yG-s3FJD)6N}5QAgQ-rX%8f@jCx9R~eQpF;Q_H!9YYgA^H|wY&y!tb~j;b zX+-7VR7yN|zW|JLMOmNyq&}nkE!qp?En1xlky5dnTHcs&+qbk`{0kmFmy%2=rqaAU z(rut4z#Ch$?vUReS>9{|0}$V_(=6qU8^*h1q|B$MGdWESXR$%H(AY3%6zv`uOblcN zA@tO2MFRPtfAR0vFZ`)VpXgPwbY0Pp9AM9YrPhy(a1Oi7Ix&*5{zD-ARPU98HAM5a zp&^ZHRu7H>)zT?^E&;Kr`6xB#!y+v4og=%+UyrA>Ka{oCHF?X^*L8Q_rKn>(>Nv0@ zhi{(GD1~ilo7DAUaKqH2eLd%lN-G)Ye|ekq_z3$CaTn69S!tx`J;*Hz3xnPy;zG*c zo4-?~wn&!X#ewJ+$RLEwlvjPW*g=kv)vL5AYE}7XZ+$#lNWi2@4fIv{f(au%QD$t@E|hODyq32OrM3uX}uV_11JYh zPf9J`P%j%)KNg(9{vK<7zg8ij=$%S2HuiPVbK=f?Q5U9b!0K>jIll9m1> zn&TJKk+TvI{eYV!PeeZBu0(o(49R5UOLK&s*o;N<3v@MPao3bLWh6ol!~|lXw%}4z z7?Meef)*0lv~3N!jPzdS^5H&dx|J7ps_VnvUpiC~O~ByRC;Uy9#oW=!r$OnD(|BT? z!?_%O2K%Ipi3gVVR=wOTKTI`$P3 zX3Je>Od+>k`5uy3>9E@8#lNV7=VmEmzB3o>c?;x%LgEaB3_wJMy?fn`BrbUs=akWB zG5{lhTjLdtUcGkiTwd>fQ$LcPxwXZEYu$H_^FaE=btQ06Hyyb=^z^CFRX}i-XSamL6{7brXh?Ssd*7BJaoYk+FW8IjTvoYl|8Yp`eC+>bdRxGIX z@bSF#Qvxx;XenfYA>{-nhB4gArJH+H$MB6h`X--ddHPH0qdiClNqtQ2^I!V;1>dyh zb^RfS^onoKfsF#6|6!Y z1(5;7OY~3p2P$jBam)G9*sR}Sfx7nkiy6gNF-h-DAc(S82{;&ip<*#XGS4xKu=reH zbcmAQ-e$yz#?;D@GEQ6i%qX!w$p;HogP;eYiI6CI0SJ%emCi|74u)G~mb*h$pR*o`4W~aC|e;Mrkb$kL!19#2os3 znM(>7&(vtTKVmn$cE=|3-CxpMOGa)}v5iRb5RHUAFT3sMjB-=#`{eC`E>heCQi4dM zhQQ*HH~~Ug30aR-N_K7VS{N4szaMb|61EUava?&sar(IFnF?t&ITzO~h6+eHTzUSE zFJBiNP1~TqM(RwN4C}O1ChKu@SbC+yC^$tGOR3M5P>bqhi}w^ieYV!urO(=NF?-V( zC_6|@AB1-cpZ)PTr_x5{fuAt~LEbobb4bC}kG-C@8pWj@*sR(0Rna`aS!J-T8y!56 zT?MpoX;~5mo10QTm7qifFY@srDSZ5`lQ=N0)@|F^?0284tVjiCsCMJ40qD;B_(NAQ z-i^{e?Q1>LUvxoT4rhFJy#1&KKGar)!i2#SBQdTD8Zr>#lER<&CIUioq`v01qV>GN zYsnbQnZa7CwO;6eqzD-t2T8h;2nL?gCV8?{rd6kT{TYe3Tf?U0W<{b>L8P{w6?Yv@ z@0J-75GZ(?zP30?r5d4SgYV8&wuFm@A|45dNf^3h_95Jn-}jPyyB~O$0kCKAY475( zyUs&q3t%VP$85R41EQhg)%0!b?PZCVM&<6P<-NNaxHS2*8rOBZxA^^1Hvkax^5VtS z?_NW^<7g+dr_dj1mOTmIf4o^#-?v#Y3STl*x`H2_gwbNT?s_2bAAz{n=p9|{iIrgj# z^*G-~7$+v$M^M36SCty%EEl+L9)o0w1kksyyMCCuz$v1TP2T#ZTT;>?wiT6w$^7^$ zP^#F_(V)O$>u$2*sybKTntIY~!b+5;0r_!3O>l*!bKtXUV3PnH(4s##Ykw zk%f7ubR2V$!Q2$luYop5>wsDVGw z6`MZPGt|SJStqRg?Kh`xZcHZ5sAx!(bPzI_^%L{jr)^Hi205w`M(w^^#(O&9X_e@8 zCw%j20K!y=2a>Uc*-8b!>IY){)9!KE1pYclWq)m4~Z)3fH+ z8~FRtfT;`(9L;1)@EmzDc}5!~w4G8;TkEo-EBcfq6Ay3+l6y89=;|sjmrwDxgD)?^ zswQo&qH-uJl~TS*84De+pQ?92pimx=;D|<}MXFCQy-`)~UTK$D79M$SdqSaf;8+%K zQJ9*EH|ohocMztGqz)>^_Uym1CAEQ2*_33NneNwd))eg85oV3l&-jtu8e?X&y!|>x zlFZO@xf6=2@Wub!1ewyeZ~uZ0L}r2)zBG>&@IYjs390#T&iEN^U0o!#^G_uE%df;a zQW3uWUJ4#;E3{@*fq0#1T}Egi#G#N30uI@@ES!(>+PB;TAr6&JIX1#B5(|;bBWp+u z=Kbw<|JYebHr}~n0gr8eT0MgV(b+DKE<)pdMHk72kX?TKio$7w{%{Kt`D6<-A;Uq^ zR%`fHO<-%<0cJNyj0;{4%j&d34b}4qFTRO9f!p@rh@+{5y`xcwmg*ujupJ( z2~0zBr9vB|kI>-mp@ARr@MA8$m!FIF6DE`W3^f1$4V4bu0%>zH4RTM(o{RSJUD+W_x&i+&K|28wO$ zW(nUXiob*XcnUJiuB5rm`+cZ-d`ryS#WR=s?YSWoTzD>kL>V;!<+}X`k2-*}MRmYK zkPO?iLy7d(N5@e*9=0!$kdUSxAC1$`L+CQx2GsUE1gJpyC6)GrS~#tq0z#p4dncRy zoxA4tDdI><`Gh-r|f(7S#kez$54$0DQ% z#l_FvD67_E1hQa-wSgoe!8UyT#CS{^67M7Hkw3d(a?NtUIw12HMAa&mP+=g$(Cgc- zB8)HTmA^#^aJU2!NclbpSfHMXA4lL8q4pgRJ+M68`+^6S5smO`m=iDzepv7AfdkHp z#PMJIP@*ckggUK|6g;!KrhuaGXQLVi2cM^L(BqL0P1yX z%O!{{fCL4a91a-;iN`*W&*sVYv9qMhvwlJ9jyNd1<^0x7O+dmiP;meh4+w9ebT*Z) z>;S(ao*pL}p^nJtsFEBCQJ-;zWOM~&+ch|0>>KZMhoNE&(S^z{sH7U6(t}d2ScGr# zZ0FoTbsWU=?;i0NP-1m;=CEm%4lN*=1=pDjB%p=FFtQ#K_c_C?aAKih$sv#&Zj#WS zNZfbHM9k{myeCHS-x6buNpSi_2e``j{l0sp%q1Oeu77nG9rFTPS+;pJyR2SW4F^#H zrHXBO{ElbqyN;n#mJmi;m;%>#Jo2E;@CAW*iMQW_bnvTY2h^j)EL0)Q2OTf1UExIR z7)TI=)7E+S6#`;fl3K@2E;N{swT*OaZt5$>ufNOjl*M0y7>5~apPs=t-HxCN1S?u$ z6J~Sel1X&KVq{SjHc&b90gy6~++c^oEn*cI-7KwL_K+)5eOQ%9-&qng7UAfAj~E zTz$Mc0reKj$=C#A>(03z?nY1-f!w^FW_89CtTuY{^B~slYS=v0)JSEzM!l7sp)27} zBxWsGYdkE0eBYO5sxp!w)2fzBTRF0ZpfptYtFz`{Tj%giWFKnkf^|471Cqk8SFni^ z3k9>w`tVW#9m5ix#NJ&pdrVA=9OylrO}!6M!~}%@^dA$mNH`2h7@v_Me&ae&Ap6$e z_9M&U;o;|!az)D4a{$v3LJl@j3~4L~4<8*m2hU!jFK!^o7T&fBWmpk`Exi|8P?`X$ z{0%XH4cFlP{)>ioj^LN-8Xix?PfFR}*^!C^F+(5czfT%LQHl)QI@}_3w6{d$VAsEh zKVaMh@#i{-?@5;xZb~v65~{uinc=f|9Iww5 z0Jy~Ik3+qGso^9uS?#B@v&Uxh6H3lsGDQL&`%DOtSG_!jcB6?_lq?2LUYu7W*pN5}&+$ zEXOWWlKG%ZKI{gde<*u1HPC7r5*aNhM7*BlBE0&eApz^0$;sconQ)k8?g)?TK|{zR z9S~ZRF`NmY+Tii+pvI0f5%S>^9C<86=A~fTL7LXJYjjG@a!y{M30Hvst^Z`vfyK+W zNN9)AA89X`w1oOsb2a18VGTA=mL*4n^V~A`xp4{7rK>|5#lkDEk*w##%${e$Mv3Vo z>3sCT)JrCnlTVQ>RN?-qtE-JuCM;9m2|$tuS7kp*Gk|KAc&jE^Y+cjx^DlQCgN+0~ z8Z4b13(Z0^<3k4%|4uc@Zb!dNJ6NrdWyP1X)P5w;>muQ1WL4QGqj4HUAfA};(pe>o zIZwV^G@T*B$TeN-K{PUsJ{qJTqht&Nt)#S>gNV>Th$(@NOPC%mZirS6(jGtdKYDmL zTO{afJB4#42aoT5DCN^dBA1aHhHNHRw)*o6n;bWA+G%J!o=ENRXCsNo{R40^_#;4jDKIa*~`|P&&Hmh5moi>LJ zP~|uNZ0=+m`s?1m3#Q5($G;Q4ofZLd!Kd4Z;@q?^2j&n?jUn&=mlQDe$ZVN;Fx=}{ zQ`!G#(_`poWG7Qgb;9AmO(&%hi)_H@%B`0CW#QaYw6zX=^-~Qz=$4xw?20gxgo+%p zQ-y3kl1UBG$}0XCpzI=D3}rz`_oM(WhtD2kLG}$CL^cgaTbg0&v{sxa6yL)W$`=Yk z|0e~93;((O8EN>RyTD-EA9CUU^>aw}XYW_f{wdhP=>cTZ>-;io_uL2d9Ivd%KcR>Q zoYD|~%QrW1ckVEg**&P>m%(+Wlzl+?&xr>aWx-u;+h!+&Oa9yKS%6E6H;PHkOm3%a z!@_Ury(Yl#OiUX0lpkI`;{T&dd=x3+S2PI1@rHXUJ1s}?kx3J&hYCLY*~Tt@2rA_i z6MwdC(mCtz)P(wcuj76WRYsGVKPdaU)?4Ayo^>k0K3x0lkTMLTz+sGgu=CID!0thh z5FUrlO4dJ5VG4&`o`_Wc|AS%wN1BC{+spOTa@FXlDZ9Aj^%}L-A&h3SjPx1fJJoOP z6ofaG4x+Uw;~ECEk8)#z)#DRIr()mxXP+{fzY(>pJ0qrSr*N+5yn3Qim`lfE653L( zWx@H+FIua!H4=jzNY>c! zAy+JykTz#=6SNFBVMy_M-@G!1fk#njTTym@t;buO{54=4nMx2Dg!?0w=D1cX-PhN~ zzrNeyKlyTEtHX-K6_e|^_%z~Qo5bH`J2!2j7$R#AA5?9pJJmF@e|x(n6DXXf5#qV} zW1%@Bd&$}%s+(h?Z#iL8+ss5iXMMp+gJL+oz*SvQpErF(HEyYDoI-x1Y^;H`@TZsf z2)gdVmM9Sb@=Rx7bhoG&e4sO-hRCy8$={nxn4xbqE@4JY zOf2O(w)i^9;UhyzcBS$orY%~VmZQ<5X!=~Wj^1s;xfz1`hwt0*qh*K1c5iC5S#FVS z7vpy`8J6BJ8#sfa6uUhB=ZuTfR|#9va_5jlU?;69{cF5!x9BgpKNU#)F3syNXm9g6D2*zMtf4@pX>j znOuAniNUzZ`Nu*D8McMVhcK7h-Q5>vxA2RL>s#`v=Vd zt8nC!g9tu@mMK?tsZWfOgN+$jtIkwi9mYga-Il1z8#2B>C3AD(^`V`;c+3w;kXF4$+sbV>}_)u=tpUliD5$D zdxqMyxT+a3k{jjvTfXBvcsB~KWsE4pK^1aF{s}@$#|9&;W6jgfVQPu2XD`Y>ayi32 zZ_@W=yX|xdiHQ>P$82g`3DHiq=IA`ia2^inlwc0!~q95?%(dK;X9$cD@V70`9 z$u?AfxSQNkO#ITVb$bQzpm(ANtf%H&YwV8T(<^_`=EgHY8PdLA+tW{6`(9hUv1~dr zuOUU{(bCcy@$>VmkaX(3EnhQJ+CObRYwT=N_O7x2Yh$Cfhex4~uI?y}miyu19>EJ2 zZsPGQTis67%xfVB@_iEMYd!B$Q@g7J*RbQ!ajM7xs)XEQC*1Z7dZ<1mtgWp~E}MoN zZ&T9x4JD;$0s#)9id(xS91;!Iq~MV=k&7#^c(m*qc>xsGzvm`0@RkB4K4+_A6}(RZ%EXB(1@38@m>Ea);7VY*&@vc^cC8~Nzlsr4haca zziF}6NLDn@HOMdY3mmNSQ`s`wqk~e@(kh$m93H++pQZMX9j7t0*iN0oGlqgKgMxz8 zG&Ln7oE-w8FnrQ9^e?CtVw*Lvvy*(`7bhks-#%|rvkII6l~G%@X*khe?mkSfw+h}` zExSCI<7Cb=XzqGZ@IcYRoi%MV<`kV|bEG$Uxeeua< zF;P*TJ>EQtx~5u+D`}-JRMDeY-(xg-db=F=&3I56!Su)1V}66I7H zOe1S+8#~CC-jq|^h^*7koO(to^&uTHAJ$<$c#zzaZ>n!$A)GB-hXo*BnPEb<0P z{iJO@8s2uJ1Nl%?poU8gX7#%eHOi>o@nfxt zxj`GND4gQfx~Kn^G+?)_)7@lM@sw-9uaO?w>38lI<=NyXBZ}= z<+a61i<;bkaQ+tZ^0OYe-F&WXlE`n_efigJ;dKAm3RPUObH8N#R!No19m#kJ_aE0` z`2|m(z8tBzn#Ih_EROEU!(I{Q=Dy}L6j*l2F)k^|Wv3Dx8?Dz5xIg2HU`r4%~| zoIii7CTJtfvn5VKUCw*j{=L!#v}Jl_Nn;P#bBSZO0BLn9S2DFOJ3AYf22!Sa_dC~% z?70x=C;9mCV5=IS~+z^H8<9X_-s6U2>qUScxQ9Whk47 zfv<15q{~27V-z2n){`&#=H|k$U*EX4eygG|sSRKa&|*^J7=8Hg;U#j-X7@R3lZ@>? z)4lDs=I@vKpN>D+wZDt#;|izM!{a2KfWOJjc7FJJfg0R)9)%$u8WLNr zX{DI~XJGGqqO_X)xYD8bycM@Vu#%F(Qtez@t^$hP_nOkk{IwCjsf;_RVYU9z1d;E@ zyb;KSm9|ow$qj5*$BFC3eQHZ*tI)2ZOuySyqoxZ+G zO2PxKvapNa06Uk8H~|<_4bT`J9i4~~!l%a+72KSRNsHi=6X4@(_&s`$-sOr0uhCKL z&?)Xr5g#eqSfzJ}g zty?K|p={{+A9eLvbCYMexp`5$cI{$uF)OiZIPmz%6ZY|aLX)FA5Mv+rS{EK9b9| zs?2x(F6gC0SC+2WSsbp|y792p^Iuv@8$5=FhN1>Pg+}D)dd%cE9rM4-4m^)JqzAm` z&CPS%X$%@!YioU6iAE1?v=+@^jyxfza)eJ)JVRjo=g;GTqMlR18;D?&7g@sF)i5iB3_i;!_|1>tB z5?t_STN4i*9b7@gl9yfF)Y_o`9CUaBww7A$%LQrv@k4)|xI6H$hXOvX7{ma4MMFz| z{t?)CF?iu!J9kdx5~>R;lIOm@5o0}h(kWnhi6!mA`uh6PL~5iXvgr16sga6~c~UV` zDq$f9I30D@;8thvq|#~W#H6I49@FE<&f23wL2$5By{)@j->N?BF%556 z9BYY6PEL*yGR**!V?BEGxExz@xQ(tLs{YFur~;UuOYxCo$1>iC*$IG800eLP`n9Xy zwZ@!~|NFOZJT*#a+Y*T}7?qisIZK>V962W6nW0hC_htnk2whz7K2ksV297FJSkTQ=3OUh08@%Rc7a+P(Zi9m|yidNzT$8 zPXt~N5h0g4_jjeMv&EBdo{IwCF#7%bov>b$!7`Ty58q!^U_Exsz*PvXsTpzLq;v`ncUW0fHF?oKr_oyH%9ZCBa7Iz$I;#Q}RYI zW4RHEG0enl1vzVZ62>#xnaRO%(|9K=BKQI)H-;6s`j|gS0&DX3=W{X;YKk|#m3*Vt zOVVo=GvJ4?f7+b+p5W`;++2%5x_&r^RBP|&Yz(cZa&whd5e<>iWwH92pG?I@Jsc9S ztSKv2Y&Ov6e}9n=vQN3#wPwQr*GgMffNS2)m(! z0#3bDJyr1&69yIp`mCm=R_GH8Jzj=GtSv8ijdsJ#wSrfpRKnSLQHcCAG~zS-7RMZZ z4S!&neNhEaGCDR!xoO<`Pu#&X31mtvq9jN`Uf%4^ojV>Fj6N)sL7vfD1hO%mDf#_k z@hxf^8u};{ie!4wIC-eEQAsg%U zY}IQw)`qt50EYbmk}xcU&)}k>BwPpOM}XUAoS11zRvH>3f$b>JyL45EF^dO*=YW5} z@d%lgf~Gd^o$473T62y^18!U%fnxdPqCzxTc*LDxL7F0Mec_IrfO!+MUbZD9oa7#8 zO_0&uf9%4o%a`wgI@0UdEGIJJT{;jeKUw3{Crb6|)Idn?#~A<(aX>4uS$iN-&{iPR z01mT@V5t-iO9B>zog*=LP>6qm6_c?2dK}dOm}w&!KSP~?g=*Ojulb0J1}hkuPmimn zO-?$n^Fnv?CgwANHB91%PAYg zHcQ8jnuv;u>gnjb3*ibf3f`bu&h)e)&p(7bUnr{43q_%WX93U<%63V^bY*EH8Z8mL z84w*CE45l7QnRjuLl7^VT|zBYC0qx57Z1OLME6)PQ02azlR%0DLHY=40>4Dq`z5P- zdl;Pe^c~RDvBz*~qeacjt9ag*j5nwEWC|M>UazaGtK8aLYm@#M9~bAC7pf5O+l|E7 z!@{11_%@KrD&Q5kf0vs|E>0n1y}d@jdvh&og6iWe_R}s3W)EUb!36~_H1Z+Z8}$48 zo_v!MH9$aQ5J9VxeDUpI(@f%Fe|leAo(ZSE913KZ2d>SOiN?3Uzeosy&|gIBn`4An zf(8#`qi{HD$#{r#@N?B`tz*Zd8q*Ma1*OIQe5s1K0fZ^zrZ@!~kEfGBPP`}}y1>JA zpa;xw?KVznx8X1{JOgpY&b{xiY^*WJ;3ph!`ov5{ZURE3-1%!ClfAcXYeF{22?_ha zzH_BaWI+P){ToIcvR+w800XR?>fXSKn<9+1G&PA*{(>m%#hGhi{?yI+$lzBz^tquS z=^*f}SeQ|FnX4rVbD!~H57(=jni_Y)Ac!&9{rbwMgG%BNflKYG;iox)tbO@%!{0{+ zF}YeeykpA;>jv)rl8V02DxH=IZUiA3;X#t3q8+w-eliCAwZ!aNi)Au=<_6;+uqwFn zir)GqW_Wsh+-|V~bQ7T_5FB8kgJ|Gt`KDz8@JJAu_d~|d5;qK{8=1RVTb18pSavHs z%Nh&ihFTeqf1StB;Jw4p5o$9^ei-3 zl$6AFRL9)hTtqr0DM;=NC`1AWy!HYaPSo83K7vKx+&z`tyPpreQZpuD+bSAJ<^ZRoF>^VOv(Njxla{^$~LNimCPh zG$HveCKF*eI)k+QFTY35(EpE+prFJ%u=0|mE-m<9)et7b%Kyy&Ugm!N4qgoeMD73n zRd;qJ{A+wR$86xWIbt4J_u`FTrEByl^&zkKf#*ukIVTfZR7;`Gid-eGd^tuc3#|UL zQcO&}_pj~x^NPL8&C35v6)a?w|9+p}F!E|HK>= z5;Cp(7S8_T1>RTusJ9_&U0=}-X!l@28h1d)<%knZkAQ2_xFq+7Z}=|%}by1N?z zk@Qe^F7)}mH{Nmoxc7|lj7N|A?6c3_Yt1$5`&mmWUls04S8dp=1s6sX{`ZS;4F5Oa zAvodx1|mpq0B(ui87?Us{gsjR6QfGlKfUa&rM9?oBGL?wA?OAQ{fDQ)WYL5{kX=<& z-!k9#CdMf?r|tbFJ(>g-Iky@UWfB`1xO4!;#A>PjiK3gm>L8;&|NO5TLy)#7q(*6b znB4!;CY54ppM*wv4B_Z|Zb~K<5-lyh2bbr6{>*^Cw2x@R4XApDo{Juxt#6qM8ylN% zg_@L9e5t^Q$qF<0Q!m~CqvHIjvr8$lRRz8kR0Ms&^9|^p5yJ#voxr-}60JiE73QZq z9OqkPJ=gqy{`(x{u?ui;HdIZ?F2?dQNo>+X#C!;VsUlf_@5P%XTTFa^7c5Y_cyocY@w^-%?Cgyy1yXOjy;WM@^XfD%Wl~93pA@7GKs=5aFR#R z0So`6bA8J$5s4!Hl|NA}by}zxDy~(>L;rDk%xcNnKE*t8<~*E%|6|Ref+i@ooA&Z* z?_eoG-h-J1()<$BFgTCd%l}o}o_b!I^!m?Rx+adU5$)JFpj!e+58MDZLf`*7NdG^V zmd#{W_L;@OmVQ1=H{o!=5Uf3zRd)=52L(b--8KKrMmpZVM*pwrjGg{lJNW-q;YPe6 zj2Sz$;QZc9*;%B6ZdB-w+@y!9Ge+*p5&E$)(4w$j`C8e&Eq5MbEjGFEs8+nOB zZv_Q-iXc3GNjL?~jqKE(@O2BL&ElZy`bhJ{;}0Du_eO4JfymCmc2h?0LkMcgcQKbK zFZlMVj7A8C&oq%7iSdp@A?G52-VaixAG)n?v=^*3H{=&CtXp#=>2a_&8r&Cvjx{mhBH>fBq2K7ppRz1kP)CF0Z6TA%B&7~C5lFfNZOVL;DW%i*gmm{P z??`2bm~dZ%jw#gh+R@WFQzyB5yr;T9MQMWy3V^;5Riv z%e<)m8ziY?nQ(j>0OHqf5S9fj9BG~5m_!=fwPIjO@ND}TvsW*GMCVxmbncWuQ|M%I zEp%B5loCRJhQksr&ju)2ba{H>e3KzTJjITbN!gjhVacUxgBtjusC5>40W>X@AJ>m` z9~fwou7V^Ebr=!%E?RBL=`MZC=UPKei=eT9t}vD(s?hUgzN{?uq@0rmwHQ~=Pr}2T zoA;CQjdgTD8g*Z`2480?XgmfjOwoyKzWq#^h${;qd`(_C!{KrQfdxE9hqIvLgLpK4 zgFz7P%z97EwMA#sWGdY~%d0}}(ZZ}QVHQ|(c*LSFVP-+=ah+WmUviBdx2pzKgc;Ie z0OMh#E{Wyjj{RIc$KoRq5O>(^K_l1Y%L|&^e$Z#!5Wwa80XQyNmOr6En$*a3L-;6Y zs?<9-LCgXYod!Ef=s||Q;TQPJcb0H{Pj}2nZA2|29n06qVrk zb7QVU%-MDGFQ*F|!k*Nz@c@|xG|h&;oJ@Vakq$Lt(R6NS8~bOU9cK`Wg;p#ROC!)( z_y8>v+KCUl-?rql-*$e@+^{W#*54ECpHut!sJ!oIS3on$JvER7{$HRj4DXjsSV)Vs zCh;Jr-Syi6q%SbhdUEC?P)ln<$#ZR<;HKz>BqBic6_C6MSw9@)JY@@V zDCm>raM%MABv1r2i@Eo5hke8Bqfay+vO_m1G+IhR>nnRDF$@SjI-qzeen9L9fdw%R zQ1dZav(cva>AcILB_NU)HKw>YK03zP4FYh+Bn;m{zX|nCB|ic`%I(1=%R7d09Eq3d z96C6G*&-a!NGc!yXP@JwuVuxdn(;YNQih;vrXUefMgwzCHvON9~;VM(Ondw^kUR2PBx1s6w7yZGn4uKBPa=gUr z^a$q9Ku&s9=sOA!2+HXuI~aP+(zm`)geasa0i5W!SgQ`V3wyv^Y`eo|<=PL8mCgH7 zTJh@XOk5ZC*|pg!Q#$O(=ae|U5))2o`;aP%n+93+3=%kpM4GM*eLf+ZMyA3V6q#qw z2lC1D1uV?$MXG%uJ>pz3bHr|JuFNF2NyUFFPI@Q6k%oGC*S2gsL9S|I=A1R{AeKG5 zBRx3Z0vL^jH_uZ|2(~u(){U$hH)KXKY5JGf%d}$}Wl6 z3M&J0#;sf{Xw1vKKa8DD>rOw%v4v6#DX~R4EVxAG0)1$oV>B@8aLBBSe|Cab!78 zK;RtocXl~u=FZpXN`VzWwgio;DZ2F~{tM8_+Bc_*Km_enpnKGe%Cdg3zG;?8WG0UL z=}odqwzTUSoiNq8_XdM{p%oWZw}uM zMT!773P=E4P8N|Fw8?TLWy0dA%aX=}p5WXT*Fw?*+78 z*Aa}v1TWu6emYqWO4i9$+|mUBuJU0Oqn;LQp^0^mC2L+7ES5pd^i@5j%ZQ5gANoe+ zRXM|4$(;xQvMs;7-7Pxk3{A7p*2-G)u;&fPQyb<4z3Y;clzNUS$zvzz(?yHu-&)&= zHQrOT6{9LV;mV}G_v3B*Q7+;=X3*KY#}BYOL`e?oebN4cOaf}JMN)dx4PcB_P1l^F zZ~Jjs7H= zIa)h8HqS6V>Wi!S`6a8~xQTjYi7`tb+Hn8r#+w7jz)FSZQLTBWqlKc&v|pk4D)i;D zZ`FxLTYCT1NEohR{Y>lT-|FJGy8P~JQi)bKNC7M&T38<#KYCs+wp=B&ip-E=MVaqy-ThylGfL~>591IbTvXK%6EfwBr(~f zu#0$cqc}Lqqj$MUYFpFLNTB;9tA%(!i$c!zA2Y?YLjvN8x@#&*Nmo4h3hHJ`Pi?L} z2VqBZrfUW;M2uLTp)bMHfKPWC#;mZ67#jJaNFi}dT1iRCakUxqzqk{Kdq;9r-%?TC zeNh69mZbWue*a`ppaVH9)s=%!l)U}PKM;Dh7v)Zv(AaJ3E`>D}^=A;5}y^umYk0G_4lq`J|^sgLG3c5i|fQT(Nsm$`voU9Ph4o&rO$JAL$z%{!zQthsiOLYr%VLebsqdAITB(i>2PFZ^;xND| zdJ2oPmvSl&(LH?OlnvA2XT5o*#zQCj;CmuQeOQu|Px9pb4)AeM6M_T_hSfEr`nVfm z2K1nVucI%*M9GumK5{S<+Z;Y{www=k1*n>?1J72k8XRrpvT2GM!@>#w3L?pp#eATb z;zP8X!&S*9;p%lkXsKL--ydXA1aGBK+=<@&h;*`EdljJO%25 zI*tie05KpJM(C_0LK@gmaDD=X(9;0uzh~8T3Pero&?!G@2^u>DLbyK}zYb0OXsrSk z7m}yWi2nqn1IOeZC`V9lzoypq7Fd7Kt)Py31ndH#-o3g@8E7oQM}Zk+7%7Ln`+$!r z?n`+oIODPXKzKGIh6d#uLhOYlLJf347^DFb!fjv#A&E6KDWf6A>sp{TOrV1+2a0xy zkz{H#ei62(&sZHna%twNelTm%-A@G~khP^2(lvL3-CAE)84pOn!NWOo{}1g1uUiLPBNXhN3vVToMh1M43zG`Eeq%ID8Z$73tpmIkyR{Z zfwn)7jSC&%7e2ekc7A)$^n$@ttJp)_+ac!x6iXPGES{&@kZ2?T7PG4moS4L{#y=Wx zNP{2&!U%u^X?D8Es)o_b_k^(D1|dI(U;zrD-u-Asx*Gp2o2O|JlCaG5rAz`6qUhib zm^OTnfVq8p3bdrQ&{&5>tky``B+NnS4+IEc zyC5wfQsV%#YrUiI)Z{kIWF(a8u^F^0p@T=YBH}3lt}mdElHi}ttXrqs8a>Y5E32eu zj&gf7Y5`%#v2q8ZZc?DK<^sSD!b&vlkF>Ln1X=F(2 z7)vP#&%2^9Lg{-TQv}lhu^IrNgg_H80`Cf8)#YB+)or8{-tR(p#(0WIP9JcAh){{f zG+>zKs$GrHa{Kq&KOKHMM1SwX%2dG(T}F8pbA|HJ{b-WrfKZP_xdYKBp5xD`IPARK+JLFr&HSo%1ox zfHNi0&(FxtCzrErGF?(Aw8YwzrZr%@Hd1XdFGD8+KW|h_Xcw zyagcVImWH6L(x;YznAaQ@1OD5!|!Rp^*#Gn$%XAPpkY=u!WFAff98zQ z2sLB_6~iB77i`jxXMn@)P->5mOAlDKoF}eliVr0;^O?vf*cTr%@@K143knK)K%_4B zD$jX7Eh9tPq3&UrNd0tYk}pl_cFk$y(XeZ#D=Y^`Xy-$s28I?)WH<|Rw-O=pW7FXo zzWpS%w9{&Wzu`C=UdJc*&wk{&TS1r{YEx5FmEOX_6X)l$)gPM#1MVAOwFv30=%Z%o zN5^mw>H;m09iN#gI3r=C=`R*I)YqvFDlIQB2P7;Qd~tFbND(YF(KT{iYvo!OQxr#J z|>hikqSpB1}b`qP;NlZCnQ z>>1iT;^u+;%bD^u+c$4)@@$13ct5Oq@+G8zL1oaTKXq8&G2htnvsfhTonkP`DJlQ1 z=+|$}qFX*CkI{dvGIEB^11tGUAr!jjN-<;OGf5#yyzwUw){9&QN<@3N77bSd+z4N~ zdF!a}v=Gpm>`IWi4#h#0Ghe?`z+y12X5}^QZn}a zl*TkV$B>g_XFBg^d@Qj&+q=IG=%6v?t78-Ig5YKd^Bzd@ne-<8b)^;GSOF)w$%9c-)qz9VK3PLCRY-*d8@=6(>8lOpOsdlTVdsgpQgozd> zN@2TM&OFBpykn^c_ zwSp-A`|(15M%scbUr05g%sD*)4pE5Ar}?7&&9Aqg4M!!xMTOtrGbMHEyy6&QT}wq34ZY-D z4p^wb75e9vN;s-vk2?Z1xbb#c*U3?x#`2&qplmi?34qplLYhMc+!*FmQPCZ`G?1Xx z$J|n%CxST`A@23jxNPEP=h$;REByo)>w}9hECh6DjL+4OhItM6S05YY*n#|nW z#qPk1C`6C~6hh0W(fjj%C7_Lsw>!acJjAAPO3Hs?hrYH|E4Godpr>{NvX$6vX&u#m zlGcz9p~hX6z+P;&zq|Cb13>!-d2R0uYSZ{E1kGxzFIV;Lxf7q;V+`RU(rh9g5=i3U zxPyL(Q;jP8AZODCj^Cjm8>bh%YhdBg0l`QC7N9)}@JoR|gJ5AuYzlq=s5Au}|D}CF zuY7(p44@;3Clc53aU&};x8gk@+u%wTdMffE*zPi90lE~5BY{+jm?Rg~Y=ICU2f1sw zJ6+YIj{#7I=x-GxnWTFr??a-E7^iT#H_8&?Jbbf1I(ZV;(PY$MhzIU;7Z1zz;kLb3mlIH2s@G2f}C4Ow*56l#EEjsRfu`gF2%|oxKR2&^9;- zq{?m}JJc;gt~i5e`%dM(oHCR7&eur){;L;2oAJQ!0bc{a2E!)ni1!$PJbVOvnBl5$ zuJdDGQc7}C!gl_4MhVD7d+oZ`2Jk^qrgZwU3@Hmi2A`q*P`VUZJnnx`H>hr)+&es& zuE({=@q?@{z>`o(4b4C+Gy@wAzpOh?=nOm0E@6MvoHnBD=I2GQ^Fj&)=M&rCp$EJg zO8s@CYeevY-GilInnYJd5!cdeCHFaEJv^H0{ku&>L6^d65%u` zX9egjmLs9oDeup0FdgZJ8A`3SwLD~JCh~*#6Yw_$2DjEL6=75&b`1Ifom9z4GLemh?bMTx~tMM!h!5w6$oNk*Cu!9FnD6fgAr0!HBX^@4SBsv2l8fAvTtquxyezbL7FB_0`2nivp zb%P4!i$THv4FI=3`xNBevJ_ORv=D(kUwwr zc76!P;sUpNKIBgbfMbuCf#BSaP-Y#1JXRphzdj~P(H*ZDVYwd$222X50|*-hy7zw^ zIqI?KtWUh5c~_nwj0l|0zn3Mcer-vI)T?AkfQxZkC`2D0@^Q#+2P>l(u;l?bk4Q!I&o$jM~r&9!|2qd%H z;Q6B@`>7W&Y<=7N9kFVN{PxJbjKG2JB4FlQ?lEJ5@8(-zOU3(%%<=eB1pBuxTTei! zLCAynSm*SJ#S9FD3YwfpWGf%u&&sV~uL|>lO3k@Fp2?n#=v^ozp#g(p2pVP598vxbeW%raYJAU)}G&(G(455$CZj*zMDi=2h9kaPt|2-1*upwWi_5%2{@SnRrG z1xWpURi%I`!s=?R6TdPW?h?^cHfyHx3sR#ptx^wrIy_+Xvbi)|ga=@$9`?#8^$S4l zAmQyxS!vbje_|&V5hz~X1- zc$ziuV|Jn7-l!akh@tR=p!#biy{WUc(=!e+xyNoS=7+qYm8r9u=sdpNMHK{nf6~T$ zZFY9Kvd?#w27_-k?q@3L19%FI%_4^0PxHAqZXCFfU5KHpFk06YvJletLp%) z&kfrov!f$d{qCs-;~vd@LGg77$P6OxV7@Y0inwUZZIfjv4jIjKSB}cH5o_%1-t`De1@t5SlzF4o<2#)}cT0{r#`MnOz;-r}?-aepvY1oH$FAVGx!ilwQWwu;Nozid z8awC3i--n59C{W!lxmtZ+BTN^f8`_)M@T`D77_dYpIC$wgZ<_?CJCFJ4bEuIMxdqq z6kmG)DE&+BpbU_#ajVpH11T(W>i}2wwfP4?MclG#8yNG5!4pl~UhJXoOn*if})XX*^I$ zAtfPm>cNxz0m)ymwU7)0{&i%2b|%6)HTxU4?hM?my!Z#|gb1^SuqKE->N;@hCy4lA z36z1`fpTtu7wfcRfMPri37=f>&Pa6j(#dzIe7Qahwea#-8}N)ZuZ>N_p?G-m)91`Wd3<0nR+KyXAtU`#=)A1!c0yav(ahcbW^|PQSd~_6+@q+15=dfUZKuplht9 znb5Yf;K}2-qZ{8d)e&3=?Z5!WeA(499o=m3Qc!LLEiA}u0E86yavNEDknre)bCrLem zZlao+vF>zGcj*k|AL%kvBI-tL8@C+RMt3%V<&bC9iH9&!0CSFDM~qMeS^?%*cVbk? z3GN8JJoT(p#3;^RY&*TTmP?FsFNo4ocRzDySHw97xIU;c0@7cQsxz4Lj-crlr2_X5 z0iDZ#J&}W>e#_kLrbh#~RZmgcPRJ1Cp{d1bD-Z#1}52dET zDFqIT$d|nfz;87KbS_$Qup1u-eRhti85JFO6Q(UQ)KzibC|>D=ybuGQ}=Wi@jyqZ7)dRp}~PRL1E7_c;AZ? zJ8d_UdIK7bP@=(n1hP4XzgarF?v{D)GZ`k8|9HqEsi@#Q!nGYYuVB+}k_8vKOOB(* zjJoj)j3c05d>vXxVV_V_X%IZ2_EKo|94O_+%zC+buUW(;Nc-{p@flDX7CQ)Klggr} zCmOv8DL-u;=zO(VQbgj;nGKnASrR{8XmtJb*rTyK$d&nM^KnBn^A_ad)kw9b8oO2Q z-?t@1J5RAHsX48_5iL4@>QvjW@o$$e{u*rAg~r%UA~h%V6-k`PlSk-V`fq19ORDZO zHAn7qum^s%7Y@DTSiQxYKLchQvg`Jxy$8pCKr$O_kLh|5aNf{eCe38vrJo8=(FBt) zY%>o8D+HHAn_5)5SSYz~oj>6gT%*OT-;<#bu<&XL`k*r}-s5hRAZ_h+Hyw|$Gk9Eo-s?hF zhp6DsVKy7h{wMdrrYc5T1vZC)LP4p(h>3F4^Go{pH3)7Q~psX=e*3>-{(egvG@Q}5)m}9@Y*#dOoRy$r;$EdOFPKzgm@2;9(J+q!0mHUE(;=x`x=F=qE z?btGicQ*zo+6z_OSD>XsA9&|z5tU96chC0$6nRShZ|r=2vesoo*j>eWZ#QUfP!4E% zpj9*ChB$MPlv`)6?Ni-kS67$AN^1m%xH!*7Ao``hew7@wDwWyg!tkIK8Z0no`MLz- zIsZlV5l{@JfJP&`?Dwuu-{7R$$zYA>%!Q@uagC;Nhg9mW$(S&EvgBFSVHH@5N#HlIQkhD^wMd=qAT5KTGgPtd!i!zkh$Osv3O)Bnx0Ka!wPzcymB%CP3x98}>~2$Hs5(R@f4gl^WnYXe~skg9+Um zB-w%Fq4;99 zge?N*p;wKx9^$BH|GLH&VG9;oS+M_Y5XADhx^f_R9(4PlZ8u=5SOJE0@(c7C`WPxl z2O28J*Xoj8FoXg*s`E9tjwY&?y9WY07&)+~G7p1%2k-$AR1sJ^h>8R>DnJKmuMea0 z_J9f?pUk7dHDj;Fu3J|_{~fH$TlCuD%x;9=Ipqln2w3ms_cDtXF$GH~2`zLadav5S z9en5G!;M_0#NrPQWh#5Xu$b7_#w50pX%Eam-Ri(;I9dx0z|5p+eGX%;XO zSk=ECdmdzuXY$xL*Z2P5P=ul^bvxB0Ekl~)@^Q(IUSW@-7iShj2%_B-D(u1~W9PQd7Udg9(r}y=wp)Cg$e3V$t0C+=V||oUkzg0Rd}SrOrE> z>{4-wCD!F&S9?L{uCZcrJPUg2Sr;}GOmBbIeQiR;TQibag@d}Z($0q9@NU>nyBx-T z&!ty3pim{W@eL*Wpt0!Y-JRE@%auaIhn$h@34y5$!4az#ve)yf^WdXUTwJQTg^SI6rXV_f%(n{?KW=byfJ=?FdPI(TF# zo7}5rUp+J?>GrI9zHQLW9&=lEbGKK^qNfBwwW?;H7lyU{{Nu0=5BM5fE6X(+UTt zfR6^VGOz!`8H}M>yW6`oI^=lz?tGK~v!epL&$fvuB+V|v#xctd#y?2>hX0EeH%1zd zhTn_xhOtAus4Tt(e?_Hrn+hIz1Q10V)qp*wyZ_~TsEgrJ672d@%F-IjX{6vD&PXeZ z+FMX;M%1EAZ2CwmkMAMLNec{IRS!61xRcG+?yXaQc=mMKV?DL8QQ1z{=bmyiKYFssCHmrG)2?B^o|=Ogb|t`NnJGlrPX{% zBh#-?L~eQXpm2C0({zctJ2N6bTE~h0>({S7%~?o8=Et|+rxW!{N}Q*bR2ejutYh(h zd>`RO>GUG6T=*eNFnH{cWkB>_aN6_kH`{!94~25PbN@m@9v3BD6fhzLC#<5R=L#i< zolW9zWOniZ0v17!Szs!?=uR4Z~$+qkr2E zzty`ziyO&(HTt5|FQb(BD-&15+}!$3tx!#tM7G{nZmpzBw-L=-fQdta58}6AmM^-8iD%ziBKl)S!v5VPU+f}E%jo$A~O|PFNem%=j<_2NO>dVRe&!4}7 zAz8QD!e#pO0e%6KCBfC(|VVfe{|6OzZ)P`z#{9zasReNnTyr=^c+g1}{)*X&C^;DZ9w(5TG*!T9 zEy_s7iI<txX%J^m!*S7r+_H%y3uY-Gfa4R2QGmwUBHQO#DT;qtsU-S3! z^z>{xjjw#k<njQcbfim z<$eLhH;;1~vs%~Sg}=MECrVGk&+qG@&|IB}9sF2Eb^m^`o%)nn@#CxLh-+y?*Ncu} zo^sic)3v_;wXpue@H5`wi;Ymk>~!JE(8GxqE&dAbXUoswa^SV#YiV|xYhvB7@B(R{ z+TV96Z$*in^c9dwHoko3$w$5G*1Oy0yW5j~7&jnnJpAuKLL?FDyxPc0mr8-f^m$tEaXo<@< z=Y~C>!||;Ta8uDM&f_cIkd7Fg_%eQn)W7Sv4^J;4G0`h#a$+Q8$mV`$9vQ(p{FSx_KPicVjrETVevi%6$H8(&I*wM4GUA`e6 z&04$!QfEA3;+xUlUS1FN^>1PJ^h~w15}uj>`eb}+>NUU%e2AGI842=lF;yGab%;kE z!EpCh`u%+pY$R-K=8dhbf*KmvAn0-fb?~Ya&>R|;3XKQf9zRR?V!k7BqBvgA?}(d( zBsm$GH+t{S@9BwR%h{F&=Ubkn6u~z+GzB}}S2s6b7BA(dwY|v?FP(>pl3vI9a$d`) zcgyzMw8cKm`P^~9Ig+WwvAPR!E{6ClkKT?zfQ8mWx<Uz!I#C)Ke#82#`i zZt)E?xYlUcJ_7fF;0H#ABV<#8nDK?1U50Er&RodZci!HQm6ao(x_Hg;j+`(^5{u^a z26?U7{$2EaY(Ryp(OdGwqmGqKMML1mjT=+5vt&URIcQFvgl*JIOH1wOnVByEhv2D* z2!ZgK+M1g0&R!{D^TYI~D1OHHbNP^lb)Pahxue@ws?zptC57ak^CtO`k{z&Shrj6Z z#fvZUXrAlnTpt1Pb#8`VpdK%stgNi3uWxOF&_~Q1zT$jn2WbhvUuR&Y=p#=h@#1eO z{{Jjz+vHZ|mjmCbWf$VTj;Fy`qYdO?0Zn*p)RAN#}Uuld`qqabOb?SJN# zURPA{@B24{p0#wtEIw_*a{2cVW0Sq%+KGo9wy_(+%4Lk zsey6x5Ip&xbuwTSF^l-*9nyutWQ#=>K12hGf5sh7WwjbW8WlFgeoyx#S@Gj5=$RSx zgpRR4b{%g2l=<=Fka{RZX|U1UgD&YmHaQ_&$It;7bT*dR*2k+fu3?S@BP2Z%%NIRU zC{=$hVSjNUgV+yKnhI$HT5G z75mP|H+883&_kAaRm!(>5#D(-q|WLv+oJomREeng8>5ocTk9G))A(4BhjW$f^vSHG35)T|=&>qT*A^?=9fx-`8b_};P&|v*H5`;&zSuMns7}G8 zEiX8AIZ$1*Q3i7Ii#I3@&%JS-Ig0riqxG1{poCsGn*U^wt@}!EvOABUF|0RD9~5Yd zq)X$JYArn4GVXEpy?7_$o7}VO+JJFO@z4J5Oyarl#QFZ&Ttk=IN>!?Bhf}CmS(>fk&>h5(zTAg*(94m+zMcfU7Qyo(s%o4)V{%vks-xO37GJvJ5| z5B%Chmtu+ntS`%b4kdry8DufWhbFHg(jfoMnm z&|;ClD?aAeVsNH~$72`~qQ~%ubY^BmGl#CWX`;7pGB60*+u#3q)F7as{zi3&BQ~BDcdK_rFL%6tF&C=W4Rxjq_-9A44*!UkyljI5Zo0j zehi}`?z$TcxyhlP)8e;~DW?2HvC38Zov3QcEj+rBLmv`Tcp+6Qj4(aB-@_G$35ngm zB2a;Gw)hZf@nAW}bD^oLY}I?=$KMN3WU8iRrhZcR&iwa!KV54Kw{x)J6{(N8e?WH= ztx0!#JVQST`_DBU9hZbkH@{O*N@SWC91tv8lWBx8|Ibktzs$nnsD!dI z@{_p94d^Jk)~g&f%-V42Y3I?2%y(KSe)5GXE-mihCQX3nVc{Vp9iOKOCF%qm`6Lmz zhxsrOwXWN&^!HvFyx5R9C(9)1x2w9e2-$1!4wRP}VR)qA9Uc_0)je{nm>wNQzg8pV zx7+eoAI>0s_j(b(&KR8i>4`oh_PyslmTOGdTgiQ@CZ-B8E zWeB)#FH9+Jp9_#Os0>3-viW1s5YEN+Z;t9UjAe`(q3xDeTjyOzDh(rs#__t-67#Kt zRu*6Y*xg!)!EE!u3LIC09w*GfPZ=2^FfGw5#mmdZ#dqjxS7XMdp_Ht6yl(3}m}4e{ zRbiH{-zkp6tpii-K1n{LlO*`SmVoHX(OZY_g;BvzIDBalg1>V3p1;D2SA!z+%OgRD zMg+d`MD4M{F#TC9D`u=~{)>gjqr*>h#e(@^F^(di2GCYG!K9*>guBM8 z!E@Hj317P-4Z$27eoXZH{?EtHiIju?nuK)vBA>l}kEsjs?S_j}$9C9_rw*5Xm^B}! z>t)BQy1!X?NKk)yW`?WG;{)$BPzc1!$;!U{p`h^JDZ65u=&BP-PVYwWS!5gj_xY5E z$hH|}WVXk>2^vyI?iU^&+f8J|@OxyR@|i#TclSI=31B*<#Cvc7myj0Lg~QkRcQ6GK ztW3^g?m6=Q^S3-bNzcKdbNV64n7Pth_=l$qJ8MH~kff{}crU=_oD1*p)P>p%aH;fI zNwu`JVu9{6*ON&`aHf{}zAShto6EyzG2q`*85tYJ%5Ls2H|V*+O5xK2}p-PtCz0_t(q(E&?|)sl;mG*zI&e3lDWd3kwT= z3oiW*8gfpnQ~1|KssQ}43aphQ7=km$#iB|oSwD&;KYsS?%!tF{-Pax-ngFG|dH3#d zOigX=C)3dijlrs_2k__aTfzNbbg3B`FOqSX)WGk(NJ;q~Qds1CF60>SCA@z9GHZ(F z8N4}u;oLd?7cZ{B7o)e~5O|vm>Lmg@40rE7fo}qY8Tf*pkcqg;(pjVsmY0oU95@Kv z%Twph)xQ1B2W(D z0m2&25Y6od;O=?1>jgbK4tTR-K6dn2nIDtGkHMag0@ z+Bc>>Xo9e^%fZ|lHJ)^XNr{P9uU^G*ULSfry=rMGqi>LC}MP0a9&kMdH@6lP8b1-DrcL;+n+C z_4xj`e1{X0#{k`@fxLpF80E5rV6MRH;&g+}O--!fy_`^SYl`8c`gmQGd4Tu!?es=o z0FL2NQjUx@D5AB0>HAG_> zy3aAdVFe%ddMl&3$kV4)Fk~BdagTqJB~kK`S4D z|NNec?|n@TKgKOPn*sBp$_)pzuH7R!B@5Q>DGRr1=1cp<(Z*^yB+Tn!Ne$pvKbQQDc*(1 z&L1Y~eaji*a!Q!g5EkQLz~Uh*q0t`qL|_mmuR3tTFmBO2j_<@8VMfdi7Op&QZSK}% z1?Eqtc?4i8Noc3&f}YhRNT$Ldd<}&%I&<;Jd$~Q63~Ck1KOF(3b)LC5G>r%@2{PFv z*4ZxOZ-^M}evK)b3a|!3?29Zg#I*vJ<%^HgyEOna^i)`wWr2|wsEO-8`=y#FLMkEv z4~y&E)QE1*(2%l=E&MK0ChvDZO)PfTyV;)1p}zRWj-RhmVEK9Y>vjC(Up-w1wIen1 z^7&)oMdz>5(jG4~8U6~$bk}^Y;Sfmrynca7Grv5m3Wwcf?EK7?dq;~_E?1W0Rz`94ogYNi}?88TIxk3 z1Er5(YgS;Bz*mnxJ%I6cT|C-nch?Cm0gz)C;NXWLV7~oJJ)R67mao&NPn(&UJ&>1| z4@!VJJ^>l!dzLfj9M

Kx`qepx_m8vqNtILOdUnP*_nTx$Q3rz9+|9?WW=8CWq25 zSYOCn@4ti+Gv)!r?)ToV4`)V>g_q1z``$UaWts;cO6fNEIZ{sMRw1MaGeq6c&>Mz@ z(O`jSQ(_FCYeoD1Ia5-Z3bD5wq0p}Yi%b6%>H+>PE5>SChq9(L{{EJMEQwK3JtOkV z;z!zSC3Rj2W76(yJQnq1Eg>c#pb=oharXK>kZ%a#?bR8X14sgS`uPz}|8nDHI_3ri z!*?zHm|p;SdjZD&+Eo!Dp=#iY)+|c?E(B`}&nBLE6H0#%GG)Jwj#8PMo6D%4h3rb? zeM^z+&M-$brZ)GsMmf^xz!p4!E{gUNYr{Ol-j=D24=v`9d%1SC`TO^aOiyQ7Y?qZK zTJF{+N~WS5B@BFGws#RMy6^i&fA)Xz9pQlNP(0o=45oynq-y}!GTOS?m<&Vxsh?<4 zc<~3->YoWBr*Lrv0aM$~{IF{s?wx;1xS|J?(=LNOmw9HJNJa(*6CQ0e#qTwHi(n!>0_xe-VWv+iZ3p%GWa^x1Yr6(Zp~DkaiCU15|W!(hc3_4E#8XL21j z3@{IzoXUuCGDS%BkNc423Bm0->gsyn@r)SqSNy+=t>z`cvw4?r(DvyuxJxnZ7HTk0 z(Vs%8apwe6XIo=%AHv*c}l;m-j`1p9} z%Q%;zQra{32MmIDeaHDDsgd_#)P8$={wT(uIEK^uj5BtfiL(WgWq;&ncwxn2)7=dz z2m>ehHO1*tWg+Eh+8hIF+Sd=d z@n#^~^^rsI9R81vjt;?qDLR5P8LH)^P}LJOFi5+X4J(F7GFcl2$C(~GEL35-n2Lr+7%P3c?r*C}M_8$NGMVPZFod)yxCsdK1y@~+C z4ppBQ6E$7}mI^#PJiY}VFZIDn=H)N+O%sL<(~;Z8EU?1H-!DzFn(uhR9Xx~WsWYR& z&b7zO2s{D9bKch0_8jn|>#`c&QddJZFXMMfWhEb2EwlAG3HTZ!ph0_jdA)RZcW+nL zdHnL_OPEz?Xckev?QQr_SZEu?CkVb!8edDb@ywww)DpGxc=`Dp9ypqa?G}oww_u&x z6xj4_eZCV11Lxa@v_0T~sb^-O3`zoT$QN5^`|jAml7YbhYS2AB{dV9pS|AZ9N|vXXcM)iZQ- zr{PA-6?*y$iHV7DA+>eBWO!6melWwOcU<(fAn6eWZ@-aEe*%-FogG{jPspyBn9!>_ ztv+#DYvzC%NYoqqIWUj_Qw5~Hn)|ndp20LyF0;cUAtC9eM{y^=AEBUGL@|5jQX5!< zNN(%ks6Z-zc+`{iRIbNjseIG8(+U7`38oKI^@E|($YQA<8xU;ZAN&SqU1%vqBEE)U z?0m7Ro|xDLc)0!if&xI}jI%Tq5fKU6FpGk<#BOzhbn7Z@oMdqpTr13Qw7}S}>BNf- zg41y{mg;y5q)rVC=yY16eF1)W_2$h}=#GK|3tbP4hCO$fIJx!OuB>um+G866ArI_r zZFZiEQ5f7?pgQ;da;T}Pxx&IC<+S4d>Qx#|Y=;YcEhBtF!h##l=uhE&;DG zOX3=x^{?SFd&|rvov>uRHdsAO)z>#?SAqIEHf%e3JT~T| zfws0bYqD&(a4Fbcuz3U!W5NA^1qvMHiiQ(XL6fQ6vmB5H6%L`lEL3o7WMt%;Eco;^ z*RNlrqf2Yt5206~yQHQsI*cX~nb?^(k7&ll#KgddV}oau3yX#M2n-iwp~0gx{e2P7 zU!n1K@yiJ=NL@)#|Bt&;Yx_HO>HnXhgor|_{L}&XY(~x zko|V<@J%c!%jWG}e7Gsj-~8kM=5i2j3@#Ecle~xA#q@K4!6P>I z3I=kVdbZfFh>VNZYO=~)HK54v9XA6VNXVAZbK z2hHdX3=V377vEQCN(FD5Ef4MC?bEB2#EQr8+Chy|LQ#=UKp$2HhyhNEi?@=_N*$(_ zc<5l_VjgfXFeo$0-w2>Qudm<_e!KeC@-W5?9A4@E7H~Fz{ksi(bFg>%UCPP7neXs8 zZW8zpeF4JXEm~T*v}>2{ zfy0I=v=N`I8mETb#b$!`6PLXWIsexVP*l-=O#?1hoR%GUANTBkYXwcK(tt02$&aew zPG6VgUCD1zz(Camlx$zIXn1?3 zb7ppynwk0YGs|!Y?%Np+;N;-FT@=pfnt3Q-LMz#Eey$hm1OA-6Jh79rGafnlIap5G zeg8oB*b|uCFeYvUd!}+TY=e~%5Wb4n-3uAOQ}!mrkD|%0|4TuGgNk`jn57Ep&Dnh@ z5q^P^G2iOFJ9jb~7ZGCIZWlS7-Dxs$TEv)O2DOpc(+PoAN2?-CE4%O*h)Std zoqwLDg@;B#LQ9?dNnFBKA>V{^3JMD99SK4hH}Dq7xopmX%NFMZ8IOwU*@~7h7D3h& z#XLPch=)Icn+lDhm@1&U@?R>`YrmE&3dK7;9q=`;7BsZBX7}o$j4$L7a#G;+fy?vl zC1(fx-fHJl2MT!vh3|xO#wKymY(lfy?bKCNx~+RqFhZ8+-{8K<wEo1${@=N^WmjCXTL?+5% zznvY<{6EP03#cmBu6rEb7+}yCG}4`tBH^I~K}rPaP>~MlR+NyG77!4mrMp2&KF)TiThDpF=QqAH#(AD`6!yO3x>n3N*SxNGyNO_rjaB@Ivp=DxD}9nwi*iSa4tLG` zKYvP{d5abIfX|&&*!szT56HOTQl#-*Pu6w8zxcFbQZxKPA9b!^`{#}ZkQL1S;iTYx z)#3J@!tJ3@zzak8`tvFhjXCR*@BOAAep`UV70mzr!=*l=bg7`fL^GiKMR3obFP;3t z0V*$X@oCFr4zS)Q3x;{X0R`HzVcSIZrT z^8elu^26j04-ytE*Iej;KmzXd&uvgq%Ub?^<6x^)*VlJhaN550-?#S>kFIp5_H2H^ zv&XRGzuOa%{C^j);Gl@-`0CqP?E016YinECUd*!&#WNXmw+)KR_OA2Mqw zVkIAq4oUaAUk~DOPkF}iB7JU4VulLFN;2joV3H_j>#MvaCW^fN!uQL5)w#O=Fw6ej zos9SoTT**MUo^ro~ z>)8yyuE>M6M-O%^Sy(8B_8sc0GY50mnKa`HA2j7JM9o41{&cl~Nin6lxub*`s4C!J zBvbQfO2nV=mANmS$Fo*v{av*PYn-nA9li!O4ylX~lRZO#%(N9&s74|k9emZdF-M6% z%t;y=M_sa$5$vSTg6X*y=MT!njGh#y%pvqoUwy`1WW)8K zv=H2p-abB{LDM zjpS;+BKQAZywE>wL0RA5WCl2=E5HM0zpY*OCX-aoCDsZ3a_#syx%2*NBp_nM?;QlF zoFwlg8eXDmXj5FXG@doJz+!%My{sM~;*G7p&0>2*S9b>Mel?73x=!r{VL^y$t@UYX zp|xeeh)db-g{Sn|O}>)D0mYvAH=Df5(K^`|3eQ*M%Ea(jOx1NqYYjU`+N2wQnvj{imknwB*jR zvL5QnRQt5la4}phfPF}lYk!WHn9dd|1z5O#-9P%9y_YkU!O-i6u>%@_V%{)xHhJ#G zcST2T=dO%7nyzUYNwMqTb)vR7&w{F=mVJ$9>XW}H$t64F$cI6qwqa{uJE*|-)#cU) z6BFvmo;vNdM+kYjQyVqeEk=URc?Fg0yf0iP_NDobn3jiX#8;wIX_*0^f%=m$*}$bA zL*vBANBbovJD7`E^Ln{j<67j%N_mxo%}nAe-^EXF==SkOr76ebGpn#f;|WKG3e$EE zyyB-KHQ0R?v~I9+x+!3kvE#ry@49H<%Oz9@0p7jNk zIIDDhr$;J>>pS}ICt*<}uIS5q&p)vQ6%`o0`9!U6T$LsjUUR1Du~saxROF6#PQ^v!i@AC)>R8k~FAVB(9rkU$I}q1^b9cwr zQe?e&&~3*lAU=6Yz*MH}L-y*(ouBeoMMbDYwvR(Q!&SK(2E1UCEat&J!}{KlN^({) zCG|7<-m^7@hjfB>x5VF$OjKk!jQ3V{b(eElF6&UxOt6c1KxJ(VUA0vZa)u8p1-}ErLcN{G1pi zk=iqL!e7Ul;R)5}4*4|>v0>S7wu)90zYYzIp}}UO+ZSNZxXw#sII|l5J8|b03K6l~ zU+b>Ma*M*da$Dljtg(f`AU#H2o$iyZ$?poGV?Gl^Ym72k$y~aqD2_p``*y0uTr%d# z-Fo$|Q6GeU${TSrX}D+@Zk;f7qzi?rJh!f>{4NMDqr8QtEWR$_((h>ZikXno;=W%a zS?xbRGR+v=4ioA;M#_)bCOl1J+`td?U$$0!>ZJk`4HY~_r#h**Yg1Vh4GB9ivPMab z5ShvmUQ>}^j_ztZGjK7vAawkBa7M!wA%uZ)$Z~fEjcOTF^C4(mIsYl=oMYl%_PT%? zC!MA&=99Ir!*P*7+AtLJ3dV!c=bSVO}wQs!XhdjwX)f!T) zra;m{ziWMfYkK5(y*IL8a{Nmb@@7IZs{+R8&3)prLdQPFlk-Lw7`-rh(|Ep<%@tI{ z_E?o^NDe~UJM2z+2d@@$x#fnZb_bRHiwm&h=^__D5|Wo}@hU#*_K#~*xZTTmtA6|X zw(|G(l*WqzQGuMV*ly0Ja;s-PolbmpH6DAE zGps|>t9B#1I&h-z5L=s`iia;?5-dR+o)X0(`-JtmT93(a2Tb)fi4{AbL6z8!nRNW+YNZkXj{4rh~pY6e+_UKRP_ zYD|WvX&RrUw3#)kPZJ#~9b)1&g^PH#ytbs57`p(W zjmmf$>SoU+f^Gs-FzB5?J!XKE0!iLUZODil~B8cI^Uo_YloLU~jN`$nu;26k)j;Ik)Bp4>mNcoiTH3iSUUBKXow>igrL z=cZTp=$gJ~NNgV_tGiYal853$;n!siuXL|p2P@=8s}&~W1DD>{Dp@S7ll}dBUl#Ve zJ@5AUM*A%4$whM!e0~xaC9uj0>BZ^Q|4H8Osjn$yG(rHkbah&83nK^xDcUt2D#!Qs z;BLo19_sDV$g>dW<*p^Hbo+qw3Sq9R^>r77^8=mwAW&4$w&`8;mV z9ns6Iv*)L)%BPV#LBy(wUUhhyPG6BW+x^uQ?9{VBN>7YdGHf0$s3~AS2N~;# zSnk2O;3QB??x*)XA1{P!xT!a_}Y?xsSNi*i}5D1GEW{r%!toDubSjd8xn*0nX= zr6tVSo8xFGOe}|MIeRATt?&3~3=K@wvlUs|_xja%$8V!@U5+JU;Q$Rj>iZk>xH!l> zO3rzlh#lr%)WpsO0s>%xdB(dQ-i7urT=e8*m2DS?3;w&TVY$!UB5|VJT=N2Tk z9}2f8q1a7&onzsT+}$pZ?Sk|Fg6H9r&%`beOyPRnlp(oVdSnEE{6e*&D>4Gu~afB>#m|6$_z}l1}G(!Wub8AQ!gdR`)^-c z*Bx@8*d%X(Hozat*0Z${`+VF68>KtYEwj=#R7Wg>K?0#0j2&;V=B?Ii!!_1g44dc0 zLoKWFNrz=skWtH%avS@@^MZ1LQi!7dj*43TdsEA6C>?tY_{6)>k(%*@FyXOz&+&-m zah$N~fG?d4SqN6~M^y(Yk?8yFBp$*;f$Fj7t64ST-6Zp= z4J3GL2TsSy#vKD%L#+&DpZ|6cQ$4B`9}LbosBwLpdGxuD9_Z5~Hsu@?W1HA(*JWJd z;bOauEM1TB{WB9zwx#nVtW(&a){8f zWsz#GE_>Br{yaIOIdU#}4_8?oz36?_bF9(CQ$+Yqt9xHG{_M?B`T56&H+|pPZgmDu zAL_-EHbBBgLqP-93@H+H=DQZ`l2%xUkH_9abqhB_J9{(2SKJZchpv_(g+jLru;x&x zb;A|3MXC6t(1J$I=@pV>9_=7`omD27x#QBJsYeQ*z{8jCq^gj+WCe`lUt9tTL3Ep1 zx3{B|3Kxd~kKkRYxE4uo7M!;tKvx!kTfxv{z)7Pyq07W+1es8QAelL*_J=%2U@)FcY?0F7x18(p!STv)!|%Ll)hE1xoVb#*IB<%q*C0%TFUQ7eA;9a4!y(8dlorRuPD7}vA0^fSEIwEk$p1QiHa{) zi5qd9_Uh{Bv$a2ihvtvzroN&P?S9-R?%KRKzD`TR6P+O|GmJL5^lPM#yoUoXo2{vw z?o_;&pN4z@@`ka#^*KxZ@Ml$y7i!Jk6GS|O074C7`VqcUF&UWA0b$=&o?DKW?K6Jb zXZgxw95Z1h_RC#Q>waH)Px^N>)9csENdc>;HTTFgMD%usgB>xTgMRa-y4@kKB=e!V z>lhiw>#wcPjq+$zoi)M)NXRduoSX$dG^b|wvYS&GSN;!9!$UAC=|$TLU%ohi0XXnr zXeG=!rCrU>FBJXWT-De?vi6IS4En||X!3nV6ke}i#}_(ZFqZQ@2vt>}BOho=^{jFj z9cgP}kSr-K9(3QMJFEIl3LU0o{C;TTvIX{LnpKTgQm$7s%70)FI!@=eCI@FzZ`g30 z4}BN$y2n_Xyyu90))U4?aX%&oqS+;`4`a|F}spIV{rk z@8+6=Gy5Vp&5v_$$F8n-(2n;ZP6EFNY3~r9ZFdDS;(39!o`c6&U-896ZXKKOixcM zsH^W+q=It?qPK8yxsO67V@^))9H+B@&;e9>KYnGMzw*1GAt~7;rWM(`FsKl>Jb?iR z7_um*+$$(Bq6|?(`KY_NSFb{tkx0)kY-d5;h#YHN)|rr}3+?7fY@33$io{mVqw9+# zqQ3%!J<}SEitO%X2WD4ZK%juKF2mq2ylWnnlmaBK<2hw@Vb429FFL%Cj#hN>aC2LO z-lw168-bA1%1rjCOS`(dKr8Y9v~$3ezX*5%JSI$(SyxGU#Jo+_ihB!r1Djmw9fx||9Nxq5wNOu>*H8O>R38DI&fyJXtL55kRdqluOXBlN*BboAdkU-0md&q zzb&_BrAY;>FBumRi+N!_5ET`zBG9dOsQ><5_eFBeiV`NO6WBh@Y8N7SQ!svav$@;2 zh~IYU2Dqkx8WsbSkORdOp3&ai{_|(U{I3{1I2AW}r3}6gbVH56^UGFiJ8CfYgK!R! zaKo|eh}fjFz)&7kca-$>7|0|2@ZoJ{rb*H9wQGxk6LvJYF)T~<~We}qgj{#9dmp9sMVatm0nlOSyohxG`a zH#ji87#0yh1!4-&07_cHF~k)`?KyqMMT4bQ)PFc(SPq;{@lz(<%b>%Ua68Uw!F!(d zl(~KC{A~Rk_-w}PU!W?~1(l)AH_$Vv#&F<}a9!z&achd3Jtb}YOGTjf(9BQvi!*Ts z`ukr(g=v=4&eyqZHr_^gAgbMoYF9*Y20SJ**$Y}AD%MrEX9lLG-(C^Y(w1}|gP06> zPAlZN$;LDV263ed=Y8|dlbo8fYbb}`$&s?v7QidRT%>nsD8bVl?EnzTTtrpFfu`y~ zpQ58beEY_NXk-`|ek?k89sBtEi^HTM$Z!xk{@uHG>ib!J#sPa3`i<8Ry%eapy+T9X zTEf15{c1e1KK?}r8A!0+QOjzE!WsU_$b{8N9I*&2ECxK3Q2L4W*33R*G!)qOTtJOk&wBaHQ;&aGCdMX$(mnbrTahv5(*i0*)7ls8Ilc{H5~Hcv|r1!0yJkxm#1d z{xFZ0_DH}4mYJBkIx(>F5{KX58Qx;kRGV?>X`QRU(rfU+Q~Q@_?lW$Dm@bo?QJfz3 zIJF)Oiu`55mYEnB9)>A{wraCdPamr`O_roH=w!*JoOpjM&p;~aoR~ZbGzKUIlb`J8 zt~9*o#TT2qOjhI!9~P=- zJ*6o@7Ct;yP=XnT-RSqu%3?({An<}=%tDdv{)^Ty7y|&qCk94FyyU^6 zbT%V4(2Du_`5g%G@)keHKJ;vtvs>>Z2TB(D$;hk!vXnk_&k{9n1M3Y^24t!Lv^#XN zQFrEZL5Gl(miFV-@myPE5`^(E+>4^1Zt$19pWMB06e8nPbAINW*WuxT3LOFERM(UM zOk%)0jN!AmmGda6cXw)am6hCtn;2I2G>oVtat~0)E%iOj6n4Vc7ah*qX|`th^EkNN zU&c%p8}*y7?Cu>ufti> z{gW?-iiiYUuH+ZIPN2;umwDYqp`RUbyU1rOX;aJ8oaMU-`H~h{oe7 zY#F0Zu&>YsK2|Z5K@lWkJV!YYa~{fKVUcp}GUqQXf?vSCz`tEz&0tIw~RV9Go4mBXcKz$G#I_&48tkD9kfVO;79FxYRost1}Pu z_uoR4WoiWWaQ}22%1o4Tpvzsv7XppsyZ7&>ASB>ZccnbclRnO1tA>xFWMPqIk%N=j z11ZJ45K)cfak>#jim-RJQcS}!b;(1+RZ-eqce^&7H-tBC67)TWlkVEEX#nQt6otstI zE?V@<`Dn*NMlz5UA{geKQmu^{B<+h+L+x}2o-iT#;@dP@qul($+Q70Z1bA*M(P=o0 z)$oZU@`3imC3Wq)Z`*qm3aBQOsP+^Y>gekrEdJhTeEt!Qq^~P%04IG1qD1-}yAF^@ z;;YTvKxQD+i5+VekbgG-Rm!UgalUHF9CPajbLB<*7tBd!sM_|F%dupp{voPHf-&?g z0L9#fwjhs(wPgINK%Y+Iydn%){fF@q8+t2MAx3j1 zBhWfx03kH_c`EKl-?&#w0w%SH5N1)zB|!F?Mi=ioy!OF}Bfm|*PfG*a$+-Kc9X1;e z*4U}bn13=ztYHUl{AV?j1-9Bd%N*-EW;=PWF^5%E8&OZTkI< zN|CqHZs8^-PGya0Ka_XK{Q`mYuE4eLl$76@Ak^Eg0agi+eMVcmz&^cnbH@iXEq6-$ zlNIn=@JyO&v{&BQ_U!zchuLJN@e-cZ^3ljZC@+4n zhUt0>2|W0BKv;I24C81oz0<+bssE*6KRMSgX_{l5S3k6J{K$)_Ju>TOYxe~N@b18{ z#x0s91*mrDl}?DV>W7RK`ejK*z$`ZFj(V(h{)DmuU~WRR&O32Ir;k1|6AHLkbm??{ z1)w5#eos(BWD)a;q|WbuX8@$xjqs&hp4EHk2Ziqeq=8^F)i|C#`01nSj4&VwoIokw z$cp#qu0sbN4tILa=eBc&zK6UfiNqSi)_^Mk>{b0*kfZ`GAAz$#*Am2EuwjW)RT zfa7d;qKDt5dVcq7vi02K?5rt4+uOqA%kS67$U2X!p05a7T{JicsluvG)x`whU&vW5yacI*0AhrB-=g{&K zM#%O5+hY25)TNHJeBdy$^aXG^$naK^op{js)4hUw1>`VH(BJ!(5OwEs-@AAp837Tq zR4vZf3;HZ~y-e(z3T)YFd4m^FSPPS_aB@$k6XzHy^G8axY*eT^9$|k9N}JG@0_^vX zkjR+pCaA!QwSA;^wqerq{gU3Khw$JnkXJ}y0ab7OPcdn=*F{4M5E6L?M=Q*YX(UDZ zq|w3V`@mC@xeY+kV3_sI<6pDuEzOOvEP%)`*lYWoDJz8VR3OR#bS6EYOU^SnL zWP2+Gur)sbjql>wiYeCUH&Ipv5|e}Lz(}({Y~!5Zt{&WcuZ`{Y)^{KwO4CnaQZ(%< zBOrJ41H87~jz3PX6X1OcLg@Fo0tNBDkSjf|Pa|#(a+cf%u*NVy=k33+%`b{eZDdSu z1sOM>c{{oD`ex6;12TWNJzIMZASb?Y_Nl-|Ina|#XenAvH`0ImD4;1J_v}?q3YOiv z+#m5yBHar>=chP1ADsN&cFR}uBSDQH9~bmh4-AbTy=nWjyd77bUtpw|imxgda|=uR z11oI_1i;NvAhHi0k6ng1#R-5sY({j{yF*6K7irDZW#}Qpt*%n4#x`nW0z$3%deAUe z-W?_IaGQO)rGCCwR6PKp1GpRVMYUal+x9Oqjf3vGzFVL!W22!!)FvsvG9<3=EcXxq znEVWxyWT+o8bzd%pT zK^0G;A^L||aPn?3N#ssU|Ecy#8Odtgv*pqoVF)e<`T44d2=Z zAj)*u{U#XMh(WGP>X4xNby`GNG&H*#R$0YG23;kMTi`WY0_s|zW142akTvEdyY|tP zhW>M3`=E`2kZ*$sp7|yq!-y?W=z88?@c~FbE<=KGI9)aX-uYg3Rse2%=2YIYGuxQp zJ;Lx$9nd{n%&<}BYy@@{qd0hbu?)n%_RYAw2xVgVu*H7jPE0X}&=s9d} z=$c!Axwxb1Jo5ZgR@e#}?g)G{9^jaOkkKtrVdL1Ob1m#cs#;th>wi@}2il0-mooNA zGe_luwZQ!$c#H>;v-kzGH0ul0jRsNAu>tlT5nfT5s{*L-;UUWn2rIrc)#ck+zEi|= z(!YsAeKR3a!2^Dqr`CkEAxWF$)w0U`s5`;Mtb$<~xd9AzTdp$q zWKXQ4%6Na_b$XKU^1bbT`^XX~jKFBBaT}x_uqV*~Z@t}-6;G~fJpmsy2*5Ldl>>uO zmJSJi4wZ8oMaCs<%$3^@Lq?!#{TNmSi^i?Tt{y;i%;VWRDA(?FSh~h?(&1eyeu^=T zuj9|;RX1HgLIdl`YrR)%x(x7Z*$BJiazR9c14RqRk$s_j z^i9WK+E{Cvn`j*Z$vgkZPQLn}# zX*R!@MKpPZjfaUfE2~R{`w|9r@^tu*+kcmpbed+i`p{d7^`L%Q@&O;6mF;V-3Q(Ux zrVkFweh?L}MDs00##cwf390R3OErB!Kf<|h{TFEl8unJpD#RkikP)asTsD;GpU>Zd z!QRYpf`*dnN4Jw)EnFi5f#F0$0ml-wDwR_Yg}b6aQJ4={Pc^-(Thk94e6j-D#a2cW zGtzh@+OFz>NRo)(d4_Gwccsq`_mAh=^|@U^E@8`bJK|ydbyO1GHd5W?1Yo)iKTo#< zp-Ltzb&hQLA!@NL6wP&8>UYHs;BA1gf9*09x;N|AEvR?Mxp{i`^@}{f33xfy-|`pr-v{SOMDwqMcK^edQ~{@}RE!;+l!E&Fp!Ny^-ZW;4@e-t3*-y+=@p5eo4^sAG(OPh}$K zkfr54Hay(Fc~3ya(Ol(i`Q!D=J1rSgE@QA%tc>P#5vrW~_C^1S8zV)P?pTktIpdhM z^!2N^>ti3yk#Km4*qcqHv@b}@;YeAgcNrO(1w823l^S0E{g*ET;?a1gf3*caDIi8E z`@`A~Bu7wxA4|L1F>N(Iy@_blyac_-OqWf7@MB<9DI-+X6ByALDB}!GZ)mbyGgT^2 zdHgGaMgl##hKE!3t)zEhW-(FMJkHyxQ-7gbCl2jt%QkYD6>1mLx^vq)6Us0Uknyx~ z1o%KDmpBNiz|3hCX(gaNA+pULy;@F(W38+&3+_`Fc9K@18i74Fu-GIq7sCw5B$g8! zk7M_mulG0xk2a7Rl8pR|7J%qel&tqw$s$6row-^l7;lQW^K{8`zu{uRyX?WP&mQGc zU5!al3L>J0S4q8B_W>eL)Zb2X%F@zsId%(C7ZA9%Y5lGuUnOAqbAMw>SY{$+1T0>V zM1Du$|1qeg05fMPztrJY_WJX_8GkvMzJ3@{Cwtp01TL|QasPn)$5h0LOj$L)_Gb~j z+*)(+l;fiX6p9SXocrOih0m@hR!0>W_em~ufTEEjg9b^8sH63isl~!6+ca~bVXu3U zOW83EmS;RXW=L)$?$nu(a|8pMKMLdC*%c9I0usyt&ClsN>vtg~Dtb?|Qe5&2c_Efb zckFi9fW~E%1bS#h@&{#&BnWgZVWEanr`XPcxN;P? zMPZplFP(>0(*0`;J&Hpwx5A-vz4x?i2wCzVM1^bNF!q13s2KNX4~SY4VuY@+5%Y|u z10e!~^jH{9>KWS~y}Mj@Zy(Vgt%~Blj6T-FGf3i--b6>U0|xqU4_=7;l=p?sLw(dmU7y5^U%#a#4et|bMLGX7EF7B$ zHrBk=e#$e`yZEvcQ&%BZy9ozV&u{EGu|K0J#%P~UIw*eZA5Bb-(`o-JT)V_tj%z>GHyek(|Ka^? zP8oia3Sl1K$L*3Jnr_-L@`k@%-1Dzy?1KJtV|mJiKUC!j?Z@mjO|PGVND6eh-+vwW z^KUW=1|S36*tf(DsQ7w-F!H;<^*Hy7YwR@`$V_uV3fDlb1PN4W1RDaEm80+G7i91nece(?TFSe_AmHSc`f? ze|sJ+ATbCr1G%fd#3~ZG)sv5}uRF`nC8a%~H>VQd-hyYW_Af04 z15cdb0RMlq8Mx~GAd6!;SjXU@ht4q9#OOW17?VFJ)UA{ev9-S_)VRva`APxOuaEYr zVR(Xb^e(BQmQ)*=Oc9cQo`SE|#{RA(1Z*S>^gh3gj!p$YCZM5M)$2ch{=^u!06GpC zg!}k~J)RZDgMmkY|D3CSBECcTlW2?xL1Cbbd{AJ{;DvtwJ(Pt0^R(+fe}oFk8h;0| zR#Jb3A#MfZ1XNQOVB6x1?c#P$&}LtiQGg~ZW^kb1#ZJjH$lHiz zE4SRm0Xe0!alf6sZRw zqka7Rgy8oAQ&R{B08AAMMwlP3Bnd%cdY)XC_?yzDVTUmf9)@F zep1kNmTb`6&;Zn1i|bY2G&Cd~0r?jxi0LHyzr!9(ifg*(nA)qn;KP*mCV=o(U=^cx zLdEf<$vh9Omr3*7&Cv(ib!15kG^?Z<`XG#XVoLr#NYeh+T&$9#AqWCXi+L{q&)loN zBmnAgkl<4G<8!+RyZ%NYKe>~9k>f?0x5F>UnebBy`r8cf$Fvcj(^wkX@W7Ig68;ZB zf-9TW;?tGmROUw9U0Opjru&d&5D_BaodNA_8;5p*teHX-2!%Zu8L*GHJVZoJ56J0M8oK;zFZL%WB1XwI8rhltB2wIImA&pcjnVKeR!+ z!^(Y7f0X7z zoP{&M1SFvICg@Or(46v*2!;FFMCFt>2&53~Y+l0l0RS&3537subvOVtFU(?9p6znF zK!+M*qPF1%qZ1c|;>C0JL#&o$@RWqFzX$vsndtx(D{4_HT$MHr(OG{C&WgF^0ot4{ znBxE<-jm|xC1Tpz+SA8AP=Pc9o8Oh}O44;d5egtcgWcZPF_vBOh#$ZHz1{ylpfm6D z0CMM3aJRRde|QrF;BTPSGytW;!SAXB;||8s&1o1$e=AT33mnQO&xIX#6poUqH})%np+KQ9fEkR{Xp!+U2U$t*hVJrCr!x6ZaO5RfxcgM?I*@B?75i0QK|g_V2N4}{ubP@f**pXnNbSHpQ^TH`o# zM#=)a;W>*l_G#6~2b18fXKAH+c?Q{M_YtYQVSyh96c3?sF)WcrJt<(*s@LRC^k-BR zW-cee_!w7ueSg9|2)rM}B_xozQ3n)$##lU9Q-G@mVvhwgvIC1nz+Vn9(m|`2^sXYi z8OAp+{A8)4-JXTG2a`~iODoS{66yv5#Uf&u$C_;7O68J2bv|(nk|CW@Zy$Xws4@^N z3-Gt3+3b9Gi+dP=<0`8zS9~fjDw2RwnL%eH5Kxwc{;UGv7VQ6};yB}Tz z8EpfkZBKYdNjw|n#TK5(LIEBjk#zU{WNIGhCOqfK5s66Ga@BLxk&<@HTZ#_f?cv~9 z*8ZgBkzdl4#lThk)O5}t-X>aH_Qz@&0$GG46iqH06Fn+{)Q#kyV*})lb3V-U&K>=N ziG*T`Dr`_lAQcN>V)dZyuo7N~IADP)<;PD}^aP6VfpAr1W@x(9Snety)m5$4soCvm zN*Rcv3mAgDoM|rCn_{5KLsS;?BEkTBSVyKdi z3&p(3YhSs_{1)6@4SY0Rpam$~bkXGFqp}FVRSW543s==j#S5&6Pj`%B6u=TBcm_kE|XtRGftdYr2o-c=5r8pobC zqdCCmrG-j5Mw9_fXtn*^J=>N5;*>frvv0nZ^Xse;XsCZhY0pT>eLqld#k<3Vb5Fa> zOe;HnN`LD^zx-r>(}H52;|&npeuoDT$_b5Yy+l{{VwI}Jcy|7JYm1JY9ZX-4-wt2$ z*A5k~fJq^&U}0&iM_rS9Km=WZ&o`bam5eKC zc5!iWJk{A?xemS~h?foGJp?pA@cntuTSAN{UW4g&(qTT3t(mH(Fab^%^egaJZ}3+} z=I+9)zCa5_S#PGv5iNDy>7(y<%nf<(BbSoqyEs;W5#e&kx<|7v1td>dy^#&Fh5-$R zrgVAKKW*ghz7$XL#|DDP>xlRt;`B7k6Cq?8geu>6Pnc9zhpXj^l004~btG8u(TjIr zHUQ<-$nYFW0+=Ps%&!XLR;34cb#Vck$F%0S0qg$qaCxhvRyMcyvkzKx`rcn{=-@ZDsS9a?r;aPOl1$0 z#ypjHR-5H#ck>Z!dUHR%>MGEOK)Nt~`_xSaoB#xj^pg?%%WcSWA+#v=6t}w9*tMqb?V$UHpL}K_lryc!X1Eq7s0M)YqV^EeZT-6 zZ`LOIJY3Oq%0m5t?y1qY=#?Ps)~6@V!O>u0trWUZ*ru%DXeU0L2XoQguUso;@SCt) z(4n|UMC+7F5>(YlTv=Yb6Vg^U7LM}pD{FMP`;Q2IF$_|{gKyAmh2{&H7Xg^+%6j&V zfTXYJVj;H9`M5=DI^)N6k3OSN0Qu;Dt2;^oSO8?)QZy1gL!kQPX9osF-#nX9qsO`n zhU@qPZL|TLJ8yKdiMN|3t{NPkIIjqX;~>!!{r+W`#j`hqS(?%c$Nlhad?L=ND9qX!;~-Vx zDP(#OfJl6c$Y*o7htCG;2SB6U&}>kJ+yz~SEwrf)>+2mKmbvbCN>tL(Hh2YOe@)SY z*)9FA$fTHq(U>{`TJF4QAAn#{&l;RjTS9DQ;gRmGWk)T7=Gn6C`)M2Uxlp3jyeOMW_E`khP zLFrM*etdifHB1vBzm5bl;b2b=k>nP$5-5N`>MGd`vy=D!NFH=6zVRICl=57Dk+@Qv zvaDLhqMpV00FZp7xJK^bP&AN#0)dKNyDLLI30dAH_pGq=B|al#rW{mf2#XKZ<%SGv z(iOOGB3HZfqje+xkrAH`^V4w_$Pc$6o=1}s!N3~QKOQ)IyA3%3;T-(XVL`yYY6K#y zvFPc&LBd9RNTFQ4myc-dJ;AR(lYrg8=Aq!`&>AP<94kCBM2rc80wLWzkXez@x_>Dy zenB;ADBm`p&B4`#)@fRn%gbOflI7W3&?Q|&4VPF1;ZaK_xMWTA_l+5&3Fx zABmp(`-Wc_83=x1T>o&6e=d2UcG}%(28b?rGGuVv$zWq|Fa1?*tbaG z4SB%I%#`<-y#V%!?_V3IyO?g269$$SF%tXN?=bKDE z3%>=d4xot<>5c#N&Tm~DXAS108JU6NlqPwvfttRmiv4hd&_Lx8L;8T9PldIizeGBU zo|$P`bLGRkTrSn04IYsS=VWa~+hAetZzsfO?Kt%ZD8;rgaovOIx|^Uq2k`YfQ%;|E z`dp~b64hw~Bmdw?eU;d;Q&kP+`DsGA-!LdANk)=r^<heH9aL(CBn6c5q>GN|bI{zX*z(IqC{pzqM(d_aHS zT}`Y%X5++hvY%u%1;iH-;zH0U;akOsLHy$FNU$tzN;#Z(EbW`V%9InMF>)qH>j))k zz`dhekz*28$*zd$%K_85CSdflB9%p-BeN^NGsKn0>d5&u-;Ny1vI`7JUp8DCg(5^{ zPp?!Crv8~0^fE>VdW*7*Vp@}{!woW53o?HXODAEa6sNDb^0-dOYI$OunljgD4;T;# zJzi*kp+CgO+|}QL*Ri|q_p#o)S8#~d5%dX9G{`QYrpCfSJvG5Mwc*GLwh}(*zDf&+ zFmg;NR}ze1n@eVn}7p8C1X1NA39edSDoy(JLKTctg#JO)naErl%jr90dL#+vQ1yFX z1|FzT-xH`94r~;EVh?`qD3shdR)e`;ec11HgW0wmy|t6`+s+~TZDiysVgR246H5)b z3UGfPKyjLq!#%&){%A|_PlJJrRsxnhHzTj-+&>8Xgr$PVkbH@eG15aJb3_ovk+BaX zJ;CMfC=0c&CGJK|!05*{;LicTh+HhuEzLf8A?HgtI< zA0F(dyrz;CvgD-zVFg2V*4^Qto%^V?@rO4!)8T)VRbZkX0P^hYRHSFfa66Ib)NeGZ<*LDg#qiFndCGu*^sOX4~w9(bH1-%(u@T3 z)8n1*cQe>TB?NT!p1!x03+ZM4{-nkS8hichESj87hk$*Et{=`YK@38?s~vog>mu$k z-^!eBDMh_mrl0TUP+!@ZiLbr0{-nbH=O7d&N#D?xC)1*AK$k`Fsx(_iObcWGWiFku zSge7V&hgPJ%n6OQYk_aMZ#wcV@veVNV}k+HPZAiT5Hw^X_z8HVka|LjNqIcq76T49 zVX)^9kO3$IrC{c*D+a_BezIy%=oCJ87Jxbq3D@z1Oo+L$jkM~HewJjJ6Mb$%M2w&> zB0i;p;f*@b+hsPK{B`ZP4A2fAF;;MdY=$Y&=`7AA&n{PM-;Vjh*mZy>@vseSR5 zyktqzIEHFZ;ZAvu)j6}|6Xbvse#?U^1fw#Tp;5yR{dxzA#pT3~;ruG9!*80yeT0Nf z1B0NvL>4*!qmpLO z)+~3#(GSOm|M+agw7g56p_k9g%4cAtZ0!;Hf`wb7TL(zzg4l=SgFw zZGoH(hO`L+IvONz;EQ;fPIqDVyqIxq*u-qJqL{55_uudU2;+$b4HY00zz11EErkpn zPONb}*7X!*MEaWCpgp9cetvZWK|vx`5WWRCWAN~JmYV+pZiuFRIRMt8B@`JE0OQ7o z@0`LeWzY8AAq&o~NniNA`RL(86pCL!01XAke6!eBb=*&RfWSa(_UsvU^^MZc`(%EP z-lF!Bm|SKTu8*J?&>N3Ehl96(-+&21y7Y>89Iyr4m|@qL#ZFcvfl)4qS_oG6Sc~yH zs8nFGZ|&>3IY4i9pKl-n9?-6S`HMtiAeIxI9bJ`>bhoaxbpoIS<{;Uv_v4Xy$@M3~ z2>b%i8l-7=xG%rh_*{NngyKgXd?+Fu0nZcE;v>D}W_&2hkPn6gt597`2C?fwtKpZ$ z?&^TS`AlCo6bjh31PHp|M$s2{41;9sqXz35Ox_OKffnSvg0U1)Q;Vc z6ag@l88GcUGcp6gR&w3hS1-tLms%Re0|Aes1aZVPbVnTJ@`4DX0(Uf({5~BKRAH=< zbwJVqlMU~)akEBSdAKf+9%jZyLa7Z*+s5xqExEWJzbdVdVcJEp8pbYE0pk~H5i}Kc zF%0@X05~rUO|=<51jzgz`?+soWd}fUY`8U%Bl50)!PHCk-8xC2-+-nEYX-x%_J?!J zJ~R<^0Q)2GH+-w}<8e_yj`0=`X%+%n76)|PFQa7Rg~?E0sBOTNci-)VAMqC^BJzb! z930;%f!8W9?6@@o^Dp{f#EMLXASRE~(+NFe@gw&Egt`Rm*I)nHC>Za8;|KG=2^LIw z0ph#^zYP|mC=V#N30G?vUoFr2L~1*bi+nr9t}#D3-x&W{GPD^FR={xTiQH2KQG3#R z;48*KTmXe2q0~jDkb&hzd4S}wQH|hNCBkn34+Et@J%~QgQ1Bs{J@zKOsor0lsPPam7i9CG z(ej}=n$N01p};-iDtz?v-Xb&W=DE$8PA68aK$Qx4j$u&?LnCd9H2U3XaX{L`{6W%we7Vx|o$VMr za-=^;{so0Z&b>zBaV=lK!opAY_R8ZY$%S)n1|`$9-hdbQv4Oj%A&o|6RGbIa1}!hq?oN6mkmLhY#Q(B_%~-78w=APz;CB z)gh*-@fM>k2eZ<|VMIa>n;V^9Y*=f@%+Z#)7Q zbx-gO|Ka*+si7m4Bvc*FQhW=TWy=z{@xh5}x+8(&vTjMgvwa5%5h3^AW<@-FK=#+i z9b5afkX{*>5kO?=NQ^{QJqXo-k;>xn0@W8GS$F%QfFXv-il9mj?St4=vyA(yE*jnt zpt@E+@xn%_c+4q1P%F`e4}BDZU$~9MZs=R6o-Eg-LW0NzCQFXj+B(H(Setv#n1grKIglBv=02)G}z#?L)_m?6R_F z%FN!Qlo1h;k`c19lf5%rimdFJ?PKrpoY$xO_xzs!@jS=zIPUNFzVY!M*L9uyb-vEc zN>hlCDSc$U-Aj7z-68u5JvB(<$jlxhhk-DVTD*T6G){{HY_bj&CHT9G^; z_drCbzd2?bwKd^XfI{7{NMf`Z*dl?FUmdU$Jp^JLME20tFLC`EYiFb(XjEiwfjn%q zmp&TdKJ|^>eDTUvst5rP{4?$eR7wOy%lF{v#~{QDUn8#XJj~U}N42ciD5{6(OXe5| z^z*L~jP-B_>Ew%$iK(!+6-a_q45E6FZ33+R*=>LC_@Vz;tPtoWt*OBX;4`TH!#y$@ zJzsqzrl1@$y|P4N){eVyM$R_+w+@zn-j*jc2dS zz0`DVn&f)N$>szzE}))-SXfcT5!v5&3}Ks4UQFdc3U<>RiVve+pV|((Go>m2Y^3ozMxR|YxaDeK4*X& zsjr#2o_UaQCde@k^J3E6ER);jtjf-8yYvmBB9VA?0P^&(=0dT;@S)3vk2aBD^21xJ zrrq_Of~uGHSgUVroncM5W0u-dzBB{ACx3#2d~Bwp-Ytk@qZETJsy*6@tgW)av?qSS zy*M+ieXqYg>eZ`ftvxC7(Mix5H^KvRuYkOo;FpVy)@d$IhYwFfL{iH-fm;vYdq z(WVauvZWJM0-xNrkmP~674X*3jA{bYb-;a)?GH^^mywk{@QW`1hJ2swk4zD$a!ORW zw16M@1uf2~jk}bS8cr=2fb3Z~^UB4%`#;hH%_i_^yzd)+9R~qEiVFagPzQwy)GJsg zDxhUz4_$|vFw=w!`XEu?IAo30^1 zGhvj)Y1BEdp+WV3Pm7uI>2Tc~k8TAUP*T!o>-~3TCKy_(>FD>tFM8tVSG8O`Tm;=s zPCyPDQz3dzq%~EMZg$}9V|?e(nSw}IPUWy*l+?1o{OO-7qLCYBwP>0S3^am&P)8`t zW#Bqoyfm2w;sK@IX-{2fMo)ID!VJyK)aOJ2U26fITQS1G)AB1?Gih5&?9diabNqNzAZ8gLU3>`lKT=xc>#;muw=YCZe`nlcAHD{ zAA0uw5>K6_X0KB$cl+q0>bI!C!`&`5FE6h68gMu56^HhIkin49f1JTDLJ(sZ0Nzjr z7tX~Iz-Iy(X>fJ@3DuXMtSUJ1L^q5WBCaG|kLXXmRR3-9QN%j{#N=6SK9mWBoHebf zSeU$`+!$nwX?KWj95!P^ltP=@asWA2{3av!r(ZkQ#FjJd@ScscN|7Y|SHQEumQ(?33 zf=NbhZV9s<6wXCyy{dyk$-^mcVVEx_CoQ1{@E;V$JdB;53Vc94)k1_&22gfP%zvY{VQIJu%7Hke$S*R2!#E(MsO*1*8JMP-y<=+wK z;J@TvRZ!nxtMBN*5lP*Gb1g^-mTrLq<#`mEcz_*y!(!~Z(xG%n{d=(r& zCii=1(ZR{Rp?-~Z6B3B%IxXADt#L*G(6g0wMdei4Uhge2y?pSdlb65l*qq?5rh5WF zyPyP~8GqX6OsEkH0C_|3f+IAymkL38?uDwIZeZ2h#`n!2j97TBGF`(*_zmjyfyeWt zLUdeUlRm@-Uxw3w!ZVayfd{75O91Q^oYs1C*0^BD0UPVgoa9}2G!zvf{$TjaW99+V znQ0b+MZ=qN6K2!haD@>e1CWf=W2?RZkwVo-EYP|?D!YFDrXcmywgB`_F2+Sx`9aF1 zVd8B6v9we_VvaXaL+`4M_G|0d@7Oc~S=SBVZ$g>A)(g9+H4O?BP+h|9ZA~dpBzjs4 zq5L;6NOn6c5dlJvtEVXhr;Nb;C`R|)V-y4dLG5&%E>kF30PIvqx!loa6@hj@y0u)vNNIi5uy4-`QCH0hdyhxPvFV{m~ zCN}Bg#z@~r3TQ5=8Lnyq-;MF=j~vi_4S4{B@PfM7Rlcob_o}mUjq{->hk%1aO%4Dm zIt_pV+FcReRAr&$QOGsDSg^BA-VJckOxm4Uyf5;X58~Z-2`)O|k0${lQXDET7rbDssN3IJT6$DaT z{rxQ(=r2DEML?7Q+hZWrD!xbp*?@ZX5^yg5b{=G?i@E0BR(#U8z*{lxwHhjcG?~0yQ_3f&noG?4fCt zL&kUT2Epw{NE+7>oDRY&Y}B4zBw7BgDS8*!&s$Pbr$ix(h@K8*&|m>yywPD_TYT4) zLjrIBW5zjyFvM94@fO;6*YN?Td{|A0Nm9cWq>Iga-a)z&tse>nF6^)H_6!2r9@_MT zK;fN*y8MQH=3c=I8_}-91CPgm5&!#F-#sq^?DAn!AnX2O3N24yDH@ zjxF~@FTx6dX&eTi6qn%G9w+tTmVtv9^ap5}Rk!tz;A^MI!L#|{pF0OZIiKw=0@^}N z8a4c?_OM@Q_piPr+>-NC+SFU)LOLQW03Qz_Ld-q9ewQO$Y3i;fQn4_8I+50hrffj6NTp$ zJ$c(Vc%Hvmm3BHGb(v9VmtKkp#9+YOHe9+O3Ixm{HaGoUG@ot$h^TlhG}4XmA4t(6UIj>d zq51+%|MhH7`jllJg-`FvCs2>i9M9E|{;O4oQMMU&M?F3|LRspzg8#(rH!IqDuWB;q zyHnf8y9=RtodmM+3%S%n!;qRs0l)s;2#pK*uWX(G7TBh}!UA3lcnz-hB%Hhh>1#xD zFmO6&pShnsE%g!Ta(c1B9Mt&ey6=;gl)@u4!D|j_Ta;XdkeY_z+8Xw2FdVpR4?#Hc zv|J*(xu=XBYT+`FEkSwm?>J>9XpD}OwN}g&wd;t|L<^}u*QMXL=)?S(9txX?jccq* zTH5q4N(r~D@j7jd@AyRvN=z9N!lSBlS24E=={! zriY$L$YOkUna(*ZN;dTxY=m&zvp7iZsHgS=BF1DeW5>b-c@+@y0Q3=uJSFZgOy~yghyT^5o4*L|-rn-E8|noEOn zbXWMQzuZN21k_2E{JXsb;?D0%-4441So!)a%x$nZh*IAuQ9?e9fVya1AWbI?1+)#Q z0UFyVji6F|>WNH9L zfgc0awv)%xptIoWAIIja;m6=x2XTX$L4CDk)IbBFfqLgH_G8B!^vqn`1R^*hjsg+qEhr!$C{cqqng;=e!k78S!S6zS z8&J4Jyw(-dLLY2w*320nmkoVH13_RU!~a1#`z2k`QV7_7WxqZ`4v#Dll72>J^nyZC zXXNBt*}n1_F{f4HEF~B;%f3s<#-Kly69ep`{=1K4LH z``BCr!P7x4fXp_g5P?vN6i!+;3M$mQ_1O)$wF_bs6?z^z)4x$d# z*BVI91E-BI+)VRwFu={&TIK*-1$Jf#_=9DGW+{^7`Rnl&kHEi096#_c)WY=PC|DH}7!@I_`1`#VEE%vQ&_V9?Ea-NyCI>K5 zj$=Gq-Z38mzTdiXiyM}0VrZaXr<)ZfK?Rgc9WIs1c!da*c7q4$-c0^}o3|<{oRnaI z!pZ_hWA8>jw(6#q5(7q^H};Fx^x8LUqPE+E(&I}=xxUE!qpxP%v;z^DUXd>|I{i_a z^&E_4L5vcJS_a;yqs7f?l3N#Yd6%{`D|!Upwl$|3JwAVBj>(BSM}9;@Zx-t4pP|SY zw0AsgNtgDkpC{?@%{Ou>Uz|j$%g&UiTEEBL)2r%*hzyurlUgRG!K;8|VPU-d|7UmI zy$3f-yvLkx^d;{&y&&c!O^7D4>`;#9lJ|Ffn2R6<%ocj9Rs`q2R1ML zeSM2Q$AK#69qPyC#c{|AgGz_?zjC%1)F4*VTcB2ypsrg+mN`0pqL{N@D~x0&4%sU zY>?wxN)V|ogLi<_RTvq?=NHXd?2?Vn&6SHbvy9U;P_iv!m2 z+*!<3F+(Ikf6L%)F1^xRBK*M5+QEG!nMTe|gISni ze8KmD-yF&ka;G$Ao zIDW#un(f86tlH0mBK#=jO>az1eM5P;KTCTCi}nV_b|a< z{h9pDM)TiXs?(QPtDcfXbNn0P^oAIEFRcCp3-No10yj_KWPDBaCNrNVGoByalm+l? z1;ap>2$R3b6H4Ua6X_KA;alRci}!qqIe6GA{XB;7(*q@>*Du9EX<8{xlnviqtrl!X zNNxPy;x%5$Q}l&@Fw)|EmZOU4V}s+!gG=RJ7e;_zvU<)2aSntnRTo zg+Q!jQD3&bSp&nVbc*I-rXN#T6a4qZQYyss$AbZNdC)Y|rFTLRrMf3nlZU5A9#_x- z?UEl{@;gSUYvH`*zanXq<8Ss=`nmMb;7;OhfeM~=zWEryK<=f%yf_$===YCMO#0wE zrx}fAemCv^$44?WA7Vhh4P!Rp*U14wP=t8+Kb-NZln^$>q{Wn{!y&hOE?q&R=fXWvUm4lZ(A`X=Jre+iW#+>okq5Y!ee@7N^2jVUwast=R z--HjU8KDJ}=`LUZqJ$1IB?%(B_W$OZ!aE1!PBP~{kyljdKj^%Ebmc59C94Ujyc3_SLJTg5;95kPQz>HJtFoKnxU6`8IwQ$!0PJ6G`(zg z1#4};VM{~E73CRvA|A@GSI0SvY`S2Y6qT*G!gW?Qwo*^(LamItx)st#fjKpk_g@a2 zdqJ3s_IM_=d$0LN?|ayVEBY6IA3s9MYc%+XsZ#1(Tf?rOU|sv3U?pm-zFJsKN~|j3 z9w0#q$2gek)ageiO+U`0DvS|qt%cUiR~}x0rJ;T_mm}3W-v6Lh@Ft?zb9h-BSRTtTd|?p&LU zVfoAp3hKv^B_}HOrNumbaIKqi&2%hhyGuSJ~CfK|s5AUzJ z2S~LW7cQ;JxOQ=LYb>0i)MW~s{%XWWvkGTD{qFFEBNmJEwBZ=mU6w75Qwh|rc&{4N zT^WX-{5Sa(&N@|YR4%7~A9-+Y)j%@H-<5%1B@#f^NWno3VlAx+Qw+&{B?z~Zhca_j z-STgti+m#8o3pblerUBZb`2H-O33#D(Z>>{hzK!WpS_fU#1nCm#lkRJT+ z7F|dQFwa-Ta`>Bg;25qvQ?xeCzYOZ$o(vZHIXv4DT+gBMb%tN)u;%*tc!X535wH#X z{O?IXWl{*xh*}+~+HoOZuij0?uW#fH25C9D%N};X!q9vL-ug^m6#p?S@4<4kdw^v7 zh|_{?*g2PD^+LF8{B=MKEE3d?&Gs4Ku3C(%6VKPQ>~a+8mh`eV9Ij+6O(q`!oYEgeXfJ#RR(76?J%&Ha>wUzW@59~ zOY>ga=)Pz7SYFDqj3hAb2G6Kbf_asP*U}BF6d}^7>P81(r99xG@?-XIS{u`E;U(ct z49|!;!bzAZN9~3`A0oZOvUDQb(`#{(nQW-wxz2*rR2Up>^f5SC;UsYid}k{xsjd2} zq+l~9UDnS(mxL4FywRKa-__t_s#lD+R%3bp>8~64T5Dma_}%-mV068-^m`bcDQ$Cv zBLK38w~tp;%gW@jTfmbkib1roB$x>PC86*{t`L^Pu)gf5;>+FK%ngDJ9s6)l?vtSh968gP`Rhu8LY^BlSukTTcq{IpKe-dTW_dNd+t5C>eoC|#3E#sU^eUjo^-i_-%1)#}P>8W;iFZ7)_ z8CWg4O~J+n=UXoYKAMz1dv(Mnq(dkYNtfjckFm|HRmitZ#z;NEG#CD{lo8w+$dpgW zoMck&`r1YPEt*5p$%!XbA+F&(P_j=RjnWeguoKw#yHK}Bnw+bOdx)f)0~5YBba`dE zn=l;L-x9BDakAAb$xzfTqAoHh<`HbNtJkn+>=mJ=f7dwaSxpHa4ArOi7-~%qSS3wU zieUftaX{kw_27=j#p~6<4|Mh(H`l~Zo?bXyoUpx>r+LEFt4Xj}ep9oqs^rf0d#1HU z0w){IgxI&rAB_V) zzu5c9{+#tMmTC9mux~}q*+oTF6!S$+Qu`tV^My_(6^-*%E`{wyRk}73Q!)0sy#t3+ zw}r&HZ)$q#R`zJ;?o*0?+pfH)Q~vkL{za^)kj;HJVQ3;Dvx~;st7M1nWE6d4gpp5E zvMO0!)6nmC)*Up$VxQRrF8h%&;Dip5t;Rgb9;6-`QywJ11V1h1E|+#~ZpMr~b z^b4}}mS62tu0EhHTJ0E~VC;&K(ovpv6N59cx?%K1TmO%re0#U(x539Dn)(gZo!PSg z47Rk*7XL+R5^B!6g^hErdvC7e#XgDXuam>WpcqNX9Xy|^fmMGFs0cMOr0cc}66za& z7#Pp*9QvmgkB1O*w4Pch{YzVaoYptC&l7Z9l6#ru##NS5kY@d_Y!gEiJWtb`k9aQq zVTsM%2tvo_O}RfAA!fQWr;ZIhGy`$utYy$mV!Q4*X^0t%a**?g&3*w!_a#vfxf~TX z1A}9w%v6U-LA`;4E$MQc(cfjzo8G{y^00&c>%-q{tEcD@oxk28&>foR%4}^PACypVpXA!H-~2dm4#hvGAog-R_0KKY*+fPU{+j*zLEGwl z~y!9Fhq$h}er52Kd&>g^p1x1eE2z&eA2O;m;s$O=bjO^B}TZ?!@BPo6S9q8F!>R1BF)#!I)zd(wNBk+l=l-=8iV-Cr{T9D3ym@3r5HYX>F|? zt(1ZVW#ZeIq{qVfkiT~$l;Jm6A`8PQA1{jCtuSwhCX``E%HiQLOqzV+Q_E6Mld&^`6akc7Fd39)4Pu!Bh)p6D%$me>K0&WAFlNY zgq&U%V;7Z?mL5w4Z-d7-?;jnn#~M9Strq8DwKj%!6`u4vGBbEP5{b5a=@3HH$@a*~ zJX)Ed4Dt)8LKf3P8Vqp$&JUTJr^Tqy>FnDi;WtJ1_seTOsY*E2x~s8MO}3W-=Dhu2 zs;J;;5s$uSzXzj-y7O(pkfq@_mmHIOWp)>LHn5@=w6C%5SJR5^wVBC*E4rZ6o4={6 zsTmG`ev0{~#k(H`zMZoAhtuts(ToB>VtadgyYrEdlNUZblcdltF8Ibq9E_32nmq$Y zc=4C--=%?>hk}xFZax8kx=G2&7+Mwkc|;szWtT9y6YEqf7NCQip* z>tX`z>$h)n6s7&;E}6xJ#PdK^Pk?c8aV;!nc<}y$nGi4H<3p2@{ucCg!#5a0Vj|uk z6_pqGC9osqP?ef?cc7uN(Mm;&Rbx}RsBr4?)GAs&yG}oz>=_82@bxso9*WMWZ6Xu zSgq;hGk4jkvUE!+&Ye4_6hQL!$Gt3rYKBVpRY_0J3cV5Qd@VF`VYa7`q@kgKM%eZ0 z);bPn-|e8U&j7t(7zR6ViIr7SS2wmyd0cC*r?9TXZgh6-&h6WeA#d|qfdzq$?iG}F zF2h27AZvkhD2^nQu}P`^W7mF5(lu#)76F0N_X1LSM)1rVBmtJoE$N5{dGSt?x%=T0&gX5F_=r|oOc;vPT8`}egmekk~OwN!(@8Y=$4>;`r{7`w& zECBn+eD*8`nn4nHvoJ$oT7uzN%YQQ*Jg#wb!=PUq7%^f3%X*2OT^dOGVa?mE<6TFB zXqWXgv$P&*>buGGySTXtz^Va+17>o-!6OGUdzS&X!>~Y@FFY`2%#j<>r}T#V5F*&O zx95hT4=;d%bw`Jat9rRb=*W>HzJIJ5<3&7X*HpE%jzH%T z3tU6!%%Y~YzMc$$NAXLkNniBr_kw4==K zJ&}=->AjX7e1$>pil{Zq6&)Qv2@3x)&QEX1jePb^8}idMT*e217b%U@M8n;=uC6XT zJbZYquN!W#T2wToR_;89b#olMzdGHKxu|~?GLzv*4d+SZLxlRj(R~tPRr7wW{OCbXrOy8hi^MoNc zJo)PGp8eTcv!kQQ^X6Y8FVpZ?MD~x~TD%DRi=-Y##?_-OCst7`&jHu(sT}mUxK%Cvm)wQ8{S7ARMR=Z zDlB}aqoc#(F%3Qab7-M}8|e4{ZA+1N-uA&&uB+U>eSndXF@%nf6kGUXo{(5pMhoCw z>D#v{kg#5G?@OFbD=DD?5~_veyx`n;INY*Yel{iItinZrS)%4MXBEpAYzk= z{#0BjX8?Okbd1Iqsq6I%7cV9!B^_|-ccR9^Rah=uNK8tSj9Y#ShM?B(aAuG_VB?dL zlKMjLmM5qt%Yxn{K*U%cxXJ){&5iAC6A)rJe6o^~2ljR*#7t@*AHt+tGq+YgroMll zJ|Xy^?ffwYVJ&(l5zmc>A_ju6PfXQ`6x;fBv93JNck?3hfEs)_4#$(WWru5O>yXz5U+u<_kg+&VF0v0h1a4Sh zIx+HA&KJw9*XEDj!#jQd(%ywfkfD{s1;%5(aY8Q0M??voVEoR?tUv$UYBGp2g#Y`) z2I+N~VHAl=;P1qQ?j#M4R3ltiO+O1IN0uni{H4%PS$%!|WYM(zA8)QTjT*xV|N9f= z-YV{{+YI0DExJ-fJzEb>6N-u}#9=xRR%6P|GwlG+8-oJh=UG0E250yZz}b^NZah%o zrnm&rYSQ08smLJOKUaqJ@^&+GQEn*bHkeLuJ&@S~uYwuS>hJ)~-@iX;Fjg(T<4i89@ts1m0*vNbb-jG zD!h3N@?Sn@{v+lo%sqNnQE>)3(7`%h%(L$0-Pqi$wc=$K5V+f~DI;@;!EKBjQLp^@ zXomR$>q;_^U|=e$s*hkc5PVJU-liZAr#2>o6HEe04CJgk4Ktvqh-hiH;3t!i5FM^! znG}P&RLJIN)zH@5S47c2$XVE=>!MsRA|hpTV> z%m@o3gH}hZJHp^Xka|Q9Z~|s*(_J<6{jJ5|y=~{G`b_8E2QLSK9S$69 zIdf!0#TC5^<+8wU0UL$|n*RBtVq)f#Te0*kX6&B1`6zJC)j&ty>|Y|w%S|N2#Hu)w zqety&b4N||%AM710b4FK7o@15pvJ~$*()QLN{}AJ4cbKtLe2JrHo^0LaS&6CodW-= zgGr?)UMb~p`{}uzo@?eU>BUhD+GeDgVPRqJ+rgYwDjqjr2GP8>2~>0>y}d;tQT3jb zhl7)|34HT7o7^)}y;7dgC78C@DJzt!4O* zpMI66z`w`MxfB%>HQmxw<7SB3m)l7MTIbOUV`it6hicrE%hReI&&-6Xd$L; zxA+Yw0z9mIy3km#Ht;;``V}R_27tr2kkgWRlTQ3;X|*A{|8u@Mj=&MuXyhjS<4uG( z7|V=x*qYKal3A`xUoM72l=JzsWZ~gcCx}y=`5n~MA_eh!b)2;|V`rzA9jj_G6${{ zE`A&T_Mg=pi^Binl7np+{Weu4v->m*i|%;&#q@S`6-ojBgbHyw@o~s{B>CMy3P4I# zRh1){7i?p=Ir80D9Ob(>_}zDOAkhJzXIg^JhoBghNyT&B1-UnISK$G{)2Bd>*e;gJ z2(HKfWI7n$gEGHx;X-E<~kz9~@vWZmd zLPY{=VwkVBBB}^~A`9is^q^J99o*R$$VMTqWR^}bDZ~Q&9&1^UGKCtfAedFf*}uQu zF8tu4iea({icD;F%E-x)m6HqVFS@0iRWQ&EC(~-Ms()$?W`uHKLt|fF^7xB`D4Xo> z?eeUpoiX@R1r`O&2T!~?@7uZgi;zI3iza5WFh=gh*ZuWJkoi`nWQAY?97-@?xdYK) z7Eur}wFbf>193iPxIW*H`9d!^)8)&=7>ZIrks;Ape*iB`3|Ds!6IraMys4mo1lf@H zZNB~ZEuMH7_J(vj4Dwu9TvQFq&DuszBxw8)8m&zD-!(MZn9R;qYRhxUT2WzlT zg*vvChp>k$C8&CpPK;6x7cQI=6Jvz*!A&J4BFxm>+yv4UnuR!M%Q}Feu5;H#W4vBv z@R;QS2b7taS*qU|9Q+bcLl_43^j{#20kNivzJ4f#gkW<2*Z>l36a5C&3{M(yJ=|7%|(kZe!jh#Jz{$QH2 z!LIHOm|+YPbe!~s;L*6_gF0*{JIvr40__i-@aZD8U@{~hKD>(TjOlegK59`>(feIF z^x)U#gDr)X`8-b(OwHMdOjHfWIb2`qHRPHP*MZ#tXJggI0u~BRhI>iM%#RIaLeG*5 zOh1b)yErg$&|~g{yX%9vP6xsR2Q0jHvWo{K!3ziQMEHvrnD|P2C>>vm*=M(n1*6q9 zUjo8ehHpQAHX+?RKl7&`&D{fN*t5uNdF+r@E z!NF26#1BmI@WjN#k}fR0imq-51T!Y!3w4(|r1gB8<@W*K`zG0J?S-~S-0RN!dww)0 ze&blzWu@cUA(Ddx734dbR;GU2?d`ZZk&bb{YBMbOrr7_Xn@|KSb&M|UVz%Kd{!1N#!w>{(fo#PWYkT{T-`@ZF`BOPtMHdb? zhhEuPKE5s+ItXejmcFuK?^yC~nVHW08b$L2Vco=)l$DzzFW*6u0EFnsje$pcE#zX8 zJ%OG-!yzb|(;!EbPe9;J-Aiynm~Vb$|K@&JqMf2LZj(VN;c{&Sb$lyIjWU7ggBDcD zJi~?-RZs6YZH`#qi^bKAGRUCJsevC9bQ&O2- zAa}TuO_jh&xC{Gr$lSsq5(gzeC4Kq2(`P#pS1vpCO+bIzTRh z>I>dbwAB*07t>=jzD-?;QVho^INy@SQI(oeFa)3cju<> z)xyE}stpvV`>ucd_V9IxME920qk+WsukauV-kZcCgnRG6t5|JIRkS=w4`2}*X=z0k zlX;-IcdlPP?bE~=%{c=e6M5-_`_1$lQ_%@4q{$lkQA6?jDM|XhnwtBMr>6~cMfW@# z2N&UgZ#-3!%V5jZG&Dv$8*6rFNy~EpRbgIE5jpk#NU%MG)sqHFnC3D(r&@<%$H-8k z?NxqfY-dNC=dQ;odcns{sc9Kvxj(dxT=s5RjoxMv^74kuAUq38pGz)nvGevsf*h&> zDmsh)P={0n=KFgvLE^tMVf%TgfYgi}I_3wtFIq7pw1>cl0FpYJJ=qFgW(WR)is>%&NWgD(2CCY%18G8MjMK)_V7y3LFyu%As9p}`>~0u1 zS4NSE@snE?Xc$^|9{nD>b_pBulsUAwH7n1tZnQbax1+Mf?0spsm4n>bsd^-!jWOS3 zf$N>0`Hl$!aV48Nj)?v?NA7;#j}8+6+k#0^{V%T{m71ps ztiGcCIfGVO+Ob1h;cRb=^b|4*5K z(4oQ$-^j%**cn_+SlLQj!kx9ZdbTN}!gsLOJK{(83T;l^Q+nHWcG)m9(?Tn!@Q}|R=T-sKg^5WI1(1C!)kC_QZtPy$ zdMMgSx|bz$zQoL|C^2C@zhL_y}^2_Gwqe*OIYL6~umN!U=dfj5$wjhQ<| zrmMNiicN4G3@h?nw8fat8N4k#kjojH#_|OYr_h|4hW`oM7ClW%t5agTS)^`NK7HWJ zzgK0(+WRRp=|e?EGtAo$ik3m)Ij1yLuBVf(-_zj|w(cKIN$!>mrBDU84WGPw{Y10u zeaQ&t(ymY#cyPTf#bVXDm)DxsG=b%gN_N^dpfM}%RawZ19GY4 z5ngC-zl;iHa}O7O@pYGM%0KC&$r9b03wp^D@33D|-ulH+Do*4Vu-yJipks2!T9ui$MA@(P8l7AY1`m|^u>(;%Px~4-`;5( zO&_0?Im)1m3<(GJ*XDA{6m`mU%Abz@8n(T9HZ=hSZ&QEAL(J{gy~qrmx8z;taGZL( zh}KBv0XcCF?`97iln?vW0eCniRR6u#0oU@d zG7NMW7!m(VX)82gSz>~mN{Q@;4hHJW&n@(TFf>v=fF{?EG!K4O_!@Zcc1($Wk{chk z4b3QyJ9Td>apeZsn#Z7kYikIKjsAYRto61~(#x}dOW8vB#@kC|fQ9hA$s6^x;n5>{ z>wI08{!@kF&H@R=16uDmwhMSI4^e&hk?>;Yv%T|t!Xoq1wNh)>V#m41`>}sdm@y>* zrif{6ZEwSeafcGz4m>`~o7&au294Kq0Fz}>wfd1@E*GEI?`LnFm|$bggR_G)d!c>OuuhQ2NHy#Wc%yZ}1cbyli=}cb3stg;(p<3L~yzA(bB2 zv3FiVz4=re`p@LPtQ$R?#Q-UiF1r8dMym1cB`>X|P9Y22Oi$LS)toge(AoHy~vj<;cx$PJ4Dvxd0|9Qp!NsqgnhR8DjY$@$`aF#*MI{ajq>A( z<|?AckFi|XE0>>pH7;QHQA<%tXk**t@249xu^w9h>-$jbg6j{Xycj&^E_#Ygp~66< z6PY3__n4Tx#=+BREYaTH*$X?@9!xv`XSPB>1zmOMNd{e=JCJ+6nB;G@ig`GzdFXxj zxWb9V6ty>^b{}7V(V#59P(cz(&j;XP^6(q?$7h01`zXmkPOabrqyS45Bh2o03I(RF z0n~%vb&^guJxdy7|3vEynG$8=^S9LpeQCqMWk}*6bkuQV+$gr4u-6&a3aA$FkM}82 z?;rS+uaJ87pz|?tL&F>82h@=6-zdKLYwlG}Ns1+Pd3DhuWcNG6 zVyB7JfXWKdD__r;Xsgl_TGdrdJsZ93A6NK1)A_;#=^5b=v7Z|xOZJ`b6fJ);Y)MGB ztECEyl1#KDG?isV;bVXw;=$nA3x|&v51xpA@x&~5pZ>p8QTUAwIIhUbL@t_et+sx_ z?*l_l!!cEsCC6L2yI*k6kR#b_g3Kg*W>Ic{g(?=J4?)PuE}bPb-WslszJg?GOqC&C zm{=q8l#fn{79=eKo9pl&0qOh9*A(_|Q6IpfiYcB6-AFkO zS?}giR8?XTd+}Lcmx+n~NB7co-q#MR{~S7-s}sfmK?4tR@)^{KZb#cg2<3r5GDC)x zudWy*B2?yBA|S{wSj^{?&kJJW|Dhbm!sjY)UB zS>d+I^&Rqv^F|%h0O^FV4UabsfT1TaEVw@x|5h0Q4QlKcp|$MtIe^DJf#c~$192Dw z%STPJPh3TKlNjc-@3js99g4R_o*DS3GJ^QcjV8I3tb84?IAAryI-ua1jJs9yKgiyt zt7r+oGnP611~flY7@xiLfm`xFjRzn3{d*VyUxx>js@x577$3)PRx2Qpo`6k{?Gr|U z9TOR}do(2nj~6K;#FV@>v(V2Na$QWx0Eqk2as;qn z7+QuVkbDHqPyZ8G?GUI%>4FC`!$+YfBs_pSVM7Ub0+w3;ZKrHs&y_RV!6mv7%%aV> zi4G!IV-(3+Q28G(CDei8i)gKuFtjwkc)B)*1Pu)S{odF%Oakhqq4q&M5YmRh#See@ z@Zo<>scOLqD%!_zEQ+Y}s9}y8MZN zi6+C#cma}oqO$0@(z%rptCC`~IeTVEaJ&%3o*KhZz zE9)*WEgb=PAJJJ3A_l>o<9n9*hTkUkWdnTI7Rp=5Ade#3I?Q}zwwrvWYbRk~n1WoD za-mfSV>(c^okNpgy;ae;d-xe?@2;@D^y7g<3UarQ;%kFa5SA>gR#HWC?rT==2Qd14 z>a`h~(=~$gkpih}CbFxSIxFpGygY79(zB9>H=#3$%|_^$+f31>x}<}TCXG&=FBr@b zcd=^0V(68~`*(GKFFMRrI*2QLLf@-WLTzR2+%&CZSLO{D@JKs^6hMd~uDc&RM%2Y< zp0mCz1uY@&0dms7MA}z=22vgE|Df?tziwlAJH^D6FkzJ z*TSS4nH}S!tzI58=qFu}WjFzelNk5VRS321uTtLtq_)i{ak{NFiQ zGM71-9Rg?SRWnrg(~Z=3S(CGF;Vz}0d-7ym5|VQ&p86nltcMR@w>7k2Hu1C_3MZgo z5?riPlmEPKF0ykfm>MEhu?!pZw;X4ZW)B^$dT9ql4z5f~rYTW1biI$7pK>3q8V#6# z^%(b4Iz9~kZ{R7h%{5JD)y8*xoA(sjV~@Gtk^LjH*Y7S=B}T~u-90Q8d#{Ih9T?}k zgbpy3QP|G?5;|aEg?=8tv8T{p!BuPhm-ozwx@;0mlBVD3U3LCHe%HueZKqGI>Y%Qj z;c~G3HTo0J#oagQZYGa=L%jA22FO8$T!TM>lF4Bxnc%;Y1S>^nPwC=D(_Cl@LHD>Q zY+O43q#vba@n8BEAr0mppmGtc_46uOyl0BXUV!EqIs?`jeF6K#Lj<@5Pk{{(8UKEm z`ycuPQYZ)DSmM9_e?B-Jxs)3a?qvG>Kb@Jy3}eW015mOE|K&IYV0e`C1VE^+sQY1Y zwdkup@0yN_dx#nSwtJWLb$8A_(HT9N0CwhX_F z+xc@w+vTZ6v)G=zpZB8K5=FinX5IJ#xwb*McvG)L5;AEq$i&a(}rjKDT$wVTiuIJC39zAsyyp1KUyw#bLx%*qbQN# z{E@{>JvHN*yI!W^{lwGh3T!qld0y6{HXeH&=fC6*Ua4*GblxFb`%3dkI`;RcFsq4! znEX6X_wiiP>(7QejrK{TA(tYl<4~3Z0|Uj4;c$RRC+t5SM|$1Cs2U4gZ22gk7++ogGzGfhv5m<3P8A8GeEVrNA&>z>R^ ztu?o{c#mLVO_N(xzVJdI9gF;A<;~b==Oa#I2iSO(@-KXoxxl?wk@NTA&{d!Oux7*U z_j}K+KkCul571WDHEWRKDheiPW6y7Gp0D}8x_is0s@kw!6h%=KDd|*7q(LcZq!Ezr z?(QxTq(izvKm?>?EjpBxMtUtm=~^JQXxPv4eZTK~-xz0)z4saC&pGptjun%+=6vG5 zulu^Ldyo%kgfim>yzEP(mE!WHqTe-ifTuHKIVJ3D7=r0h%^WFb7Onjh4r%rXEVtBk z-Z$5c?!W#!Kq)|JO|ZX8*RCGoY5<93e)nm;?4yM?r>Xx?W2LI8J?2E%MbI=91|4H) z-(@VtcV=w5HyM~#L(TI03Um16>L?p2%ksoW#Z~pu))j2wa850FJjcLN%7fT7pE6N*m7-0aw_zuq zXQb^Q?>Fl1Pt<_98OPNS#hU&@WUI+3vGi-Vl%cMpwQ@vs4U^HfEPR=rZi-!7ZWF!P zyEd1iq&Y3A;5rYw-VW(JKIEmWznN7(;A0w({FrL?M$dfU8p1hDZQZP|MnmfoMB-4M zHmd2;wAW{Z3GVe5f4b23=%Q0}7Y6W;sa>Ns-KOY)jzyEm{Wp5_Y@%j}$FDONTNeZR zytikUwqxv}$Qh2=Y*8yLZJ8T+FVFkgO4%)AQ{Q@wA!|+Zn0x6aC}CzUC)S|fI|D*$ zG2^5a(#gZZXG{I(D7yxxN^V5}w@sW6lDAubg`j0lnPV2tqXV}%uWjFPz@zt#7F*{A zGwtYoznj_(>+5q7XfHK6MadF3F^>FoS4STNfr_~RHl-K-CbR_KwFJjtEbi!ZscMB-#vMCDaudHF&X7BSW z`uGFn)QXmX56G(a~0P&JBC!|@}H5})%psH9x57%>zqQr4{|G(arN!>I#UOZ zb@ryxH*kwFztVbEcsQO-U9Wc*ikek5`0L)U8hd(6bFJdlgFUT7;lf_{SXxfU1{Hp2 zXh9FI>AoxCN6|qZF>W`c?kgL-eJ!XbgW4wfA)y4T_9x}&Vkf`23;}V!8>7l-(rx{A zX~{IZh_EHt%hK^KFB?0Ae9Zx!x_KcJ)_MlmTj{J#F`9;h#D@ zxgBDdrtkRJs_cx6&qZA`2vh?Atw6A!*uqHM(Ajr}wZivSbQ|~MJkf%yxkL$fUQ;C`jnSXxZX)m~I6sP2foO*Qx!7#Kt5V6c;?(lG&rB5kW)<8jLeNqyRQ{$rEE zqdTMx-zVc;KqWu>=k>Bkznjdh>G7|H1PQLmbkp6vPf~(i-Ieihi79;3-}Qaa40NvM zJR`BY@vSf;i%KajePRrg)@afGbdnyi`HVvBwZzaP)RWt#3l_%E{he3u(U;aPi8azJ z(%)(-fgykY<*S>Hq$VkcKTl0W)Zx_dN!-n@U8JjJHYtfk?x{_b0fMn2J>;H{rTay2 z-dWbwNRZS8m3`s4Yg*tH{+w!tm+vcWixMbKg9K~#*tBpXYwHuHJV!p3>jm039i^rc z`6Q$)ztfFt=3kaR>VMzQay=mG77A)*~nJT(fetzaO0o% z0)Zyp^Co{S24PpPG`V~ercz=+>QT|knE6Wdd*)r27}vK8H=UQZIhA9Yb^;u4x}4W6 z;rzDnH@{EYhi8|EVM_9(9<2p4t$3QTh#fnxRu3vy=MSuGmg}W@Bn$3jRQ`A&;Tc4z zn@Nrh4=t0zke_*Z*W>j>bX$IS!hx})6ET`@SJ*rLeJGAR*a^AU{s`mR>r$Uo2Nmx> zt*W*Yy!NdwTCDuDm&WirdBhwd=a$wUt{~uv9NEqa-0)+R`ksNFAJ)@rlxxLA%~sdD zMPzkRR5=(ASKhe{{#4))1cih7=jW<1H-*jbVeW>b%YL@c1;2{LY+G*np2Kq!E#nv~ zU&KGWC#YqkJG`?axqdOK5of5)xZD@FbY&rKh*@%zG_SY<_KRjNMCwa&YIVh^l8c@GQwkGxNkS#Dcdy8WAiR4P9P&(0ex)vDP>*PSB7}%9YUc* zVT@m6@Vu#Kqb~Kt>K8f1{_U=sljhFxqU45PH{l3${x<1Mwd1+Tz25NU$XWPOJKdl_ zZBH|2yKMdjm+tF4`qGWeo@C7ce*=gT{2Yyx6Rv99E50P1l5{~~H?EOl@?{>xvUQv` za0z}x&4FLZ1|K_-4|4|Ct5Hgbf-0S!-|VOBU=|Qj?IzRA zMsa-G&D|-I?hfNbOzR5R^?KVGjUSWglp!%~!R1XRV)0#-@fGd;d}Ooqi184ZtueP{ zh6uf&|9o^yQN-5pw^p9e6w(?)FcnQO^@;GyvK+Gx&rJ-yK>K;;J@)E(9!T;Z?XMwa z5h^h_%G!_VN&FEHofI~hO`;z?rq+PCIu6x#n~u=W2*0_t)Jcao>#Fx^D7}X1)3{#z zN4{pKqH?PCO};9NcEsM8vyFs4i^hz?vXf^C2WxMX<*?5Ji^iuDn)uQ7h94<&J)seW z19h_PHGb(OjG5w5X2Earv-ZT;xk*S8sUMf{Y?6$|bycSa;(|8%$qB`~Zz} zS|@YPM*6vZu0^qPwQ9AU-OypS$j_9ywrYi*LvM}a@24;kSSyYa}=ABjMU#07U zfh(K0tT+05z+MOk7C2Z@@g!A!_PQhkCdYP@$AhuD)q?CyEU_Otvm9C1-tV|qA);j` z%vOq7gADFYBsXg|wN72cb6vFy+$!LjCx@1)wTu zQm^*why!>jJSc5HjRmNE8zNmwgVC>0)N2M$Lj`u4h|CrY7&(;+~@JHQnOXKElRLTKL2{oLqpbIlT|*qF~6Mh7vD)c zPpo{bbr?>&HX2K#<{X$-PNvNn#{_cheI6rY!^eVZObfjZcF)>ui~1Q~f6!Mpp!K{V zW~^bjHV|raAJVx4TT!w-be?q}M6<_nRgOTkPMF+XShO##VMVfsGBK}z(dSVTc(4`R z3c=BxMs|P@bBgC~_flglHyM7@tV1RL6BzFxwU=d)eR>0?KYdYm*}@FNwAQg`L?Bo% zrAPX`0_#uP*Z7n&!*$pg8dj?Nze;>x`MOk{(yUG;BJ{v-YxLoP#jObws$C{Vsn+%E zTT8FQ5V^<6S1&wHW=xu4ajMg8U)cm4Ux?~aI8AkOn6@5XX_r-;tZv8|Jf?B{oUOXJ z9C|*Y8hEa#TeE-24*k3f=hD541g;k@71m}|pV=h8sQZYuEp7F{w)TcoSX>Khzn~>} zxJ4j^>6P7}gIT7I)gHTcV2~~rP6uv9^2iEn8?>)CF-@I-%}ophWpREVS^-|*RdS-|de1%ZAp(r1R7nk#~f z*c*2!9mO7T|I8A!7G2v|^}yT>@9hnh2jZiJxolI@*IUd5<^-@6Wxcv$I9Sw$n)~;X zxC9%I-rMnp%%`IjAUx%o<=zQU1mkzP-5Im{})&tq0yiPxCe9bw3VI zOdaEH9^*t9*MTTdMxPc=|8Y39&c+uzn?5tS9d_gQ=N=sKx^Oq{&dLvf8PvdX`7)1rU&p$rb+dbS<6w?!$Q-SV|h zZ6^l(4*-GzEI9GWgJ9s)s8Vmeg%Rkf$>TaJXm*3N^B**-NId^ehc8AihWQfa;|!4u zexFk-o?(8&%`~1OWEED-!_(WpC8;Fq#r6%lVDm|tC8mImrS%OBzw4X#kPo*hqcECY zxHZ-N^w1rYCam|zGMS}Zm2X@mH5aa~eLAin1}_lEt284n_1LxDynE55u1kM5$q8Ld zJn@{~OABtEdEi6p-z{h_4SR_By6L-vuf0Gs?RPrpPzGbHt_FR=`~)}jkaV#*o;*8T zFbCB2@V#k=rRi-NSu~91Zhp)NE6_d7k8A>)w_Udf0SJX451N;MS`Xv5l|QquUytQ5 zx+dBt#1W{dB;B@mk%j3QvwCpUPb@FfOgqg&VqAEG zE9YC;y5#i?OZl{V?H({(D}JfiyFW=|JFGYt_g*EY4`O)Ay>}JTK);gGIXQLF9i;KUOf;!y8mI)yk2E{KK?UIr8=hTyl(~O_sHfyNn)f z6xp@wj31f-bTrl2IyN0J%iy`r)9i~)+l;P%yNf?bJ)*G&(i#Ap9-M63e&r!Yy-()n zBGD7tNwcYm-1M@eM(1eBOsj_r$+$EM<{&+%)yY4cMcp=xnD5?;26f}fdyBad8SD%V zLnMH=z7|9Msf8Hz3NH#VHN4!W|5`M|q5O-+HsAh!TAbJsWAu(et&6#-m!L+MGbHKI zD*PX8Y!gLj7y57>wL~=H$K+#a()sR&jl*b}0Fyl~!3ime&ZAJRo>Ep>76IIEZ5J&T zNq-HFXa=%@57Smb?C+DI05)Y;fMstyNYW>8^}iisf<4uxt`X8gS8zf@3c`@FfjgH^ zW$a2>KK$`2vv0&)6$XH!-usQSW6p3hZ!5k(qIEM;+7(O3&Ghc%-~sGY6jaVA*`?W& zn+Jk`^n{6dz@RHM!j-YP*wYqNc>v^(1t5RTJl#*HZENrB#m>w8O>UhB7t&O>oYvg> zO&Nqe9RQP|`ib!oIjxIl)s?mEWE?fk4S~mW=u;we#xtxw^`PhguHhM|kVpXj z=U+A*1ls}p*Xhf_y{-j5;2l-`;r=nPyhekLzyTC@{}jYcBe$eTo%Ms1V#9M6VNaYlO~?ao*or@?b_;^`}wHtO#$vY5a5|@w!Mko%?PjZ$nyzXN&xH8F5e+{ z%I*Lt655DiU!%R**!F2dd=9{m&4JK=Hq1@ntjO?Mjc{e>jO`njM zwvI}oR$V}x17~ywg^VV-S{F0BTnkYEzOgHzjo*`JWQyNMYQAfy{+MaA-3=C#??a@J zXAJ;MYs+9WK6N1lVLImvMRB+AXewT`{dpUPzQ^aVP#$Iw`7C8~E8FFQe0oO@Tb_Yt zkBFSXh5!voe7e+xhPdh~&U769zFLW#i5KsaS9bvB3oxktTVKl_SP8)+IfW$SOQveex3XXnrFOYh9t7w`OkH8soXloe zSm;cFgs;<7&--SSv055rrR1ndi|=N8_rY4Us9C!cTWJ;U13L2tsw z{scOWt;w*LkeXVw5E~U0oSU0_UJIn=KvS(RtT-|ETt4Ud4|_Y3iH9F8*(0Muz_#AN ztlE}O$7`D(Vkw-gtoPd|6Xygy_1gD%J)uTN?=u!a!GSMDF+ECJg~p=sYrMb-_FBhX zo9-KAUQ_SZF?u4B$EZDfIe=nsJm7wfR`*COb#xXYxJBB>r^!bEC?rl61KZCJf$n{W zYUIZca2^AB?H+~nwOeS_##|{e$rILK)R<`Vpq-vRnbH|FT&}gaq(AoQB=LtE z2$QZiMz&APlVv_plizp4!KXD}xRz_#TcD=*ak}+M$x>O($s#MHQt5|Hw|@cqpHce` z)*$?a+2>z((V=&I+;Dyk-EP(W7+&ms@RdeYLgDws-j5broK091)~ zC$-R%6dOyx&S`!9snkt3{IqSq0nzr}1SF=ME4%=$Om%p7MqwoU%;uRyn#2KrUZqw5 z>*#%!#_4y`j>JN>?~26@l^>ZYB>b5Xjh|zJqjhv}jBS%`;S5Hk{+V>!7jE`RuqUkt ztHj4wqI9j2))=3U@J^IAn2kHlFxS~P=RxIc2$UM~eV7=DUiVV0xAI8$dpn(GdU9nB z6&F|Ow*tLd;y40jwUM5VV)cXeZm4!RGj0CO=U)G4Wr0%F?&J{YiIi1T@*OOzwRVt) z>moO^4k9p9Noi@6h89rn|IzV6;i*7sP@f5eu=A#9{~?nbagRQ;@&EH%nk4f8m#78F z(00k&{4=8f8318lP8n4Y%z-GS+uWXeZ0zf&_J#0ysd-Daf?B)g7Y~Xv<)iNl$i`m#Tj3$qK<%KQ0s#`fe>650K$H3e$a|Co_e1nu%63q{eT!oqcwL|* za9hjFniXm%kAd782Rl@OnpSAgV}VH;e_s zBcQCK1!^TAsaLD_MoICqvtz)^Kqgem$=utdGktcR4_w~b_|I7vw2;_K(lOmx#B0ka z=QJQPYig}n5!M`N0D<1gbzXwL(0ectH>k|yUnUz58RohKa2}4qB`f#P$Ds7Ey5UQf z1kKgk`85hMzWe^yGVAgk&)o@tw*qhn1Ox$b&}|p(T^W$Qb`Tr--9iJR^?G;((EFo( zOiK$#8C9U1gSkf6YS(o|lzk05I}R|(z(6A) zCZ3*g1p2jG;SUBQfyxjlx=Jq^z4mDi4i21vj|19U-}lLPc!3o9+qZ9M65m^lzTIy2 zCnO@ejRvGak-&E4xEcts)zs9U0OigUiSI;WQ+}&| zrnof8yCD9y7Y!?dJftf`V60Ww%e1{t4F|x5%d5)xMD(}9}mlDYWpBZt&EL->w(ATqtO%6h9O90n_JxDSz8`l0SZ?avoBP? zH_?!q($c@tVXbWv0tRYe&6HRv?1>={TOCL>oo{mA`eRC+o8fo?)L$B57gEyF(@jc& zfz{YiwGuAiZItS>Dfe)F1TGYCUboTolj`+s?ssRcI`K_)psfvSZxls^0^O$ zu>VNg?xD2+I};!>w@E;jS6l;K9a4{981^v=WwS6cf`j>bPqc_l2#p{6V}*5sRv*9e zyn$2*X86XP&lT0I`=*U{x{8m`s7^%(%Uh2Orw1Ae%Mt*hqmk-NKC9m|m6*?U-FvUx z3f|vpK8K$Yl_m~DKdhfw!*nI(rO7V^#R$Mvn7%rvHQDp~tQ^M)6kTA_>LH%0hqBrN z*(s3nq2sz*0l70uiw_>J4g7{2E~2P+U>bQ*GM&r@sB_5x5ICr@bOhzcxyw>@_Wch74_N z;9`74q@c61oyQxC?hce_;KnLd7sM*=I=`MJwsDbCELHGyxMUuM-_HS6>YslS0C zIxn+voEd2HVL+#b7mpoqjF{Qzy^m8*%?7VDY$t<*u5Qw;{$OMRxoydzu2L4??;7dJ zcrFU38}&%%*S!rK$y@ zT~1CsAll{g*dakVAGsl5grY-KzrXyXXCY8h48*$8sV8G^KVrehm##U{xCbJLnkXlo zmXgMF5f!Y!H7*AUiQ`Txz#MBcY%buA(=sshY*sCS)woos60Ev+ z{#8)69Q~9Bi>p?H(^Cu_D!p}Q$d6jCH)q+N3*W|-L}I|L20~0@AO%H9Hc_iAu!TY` zQ;E-%pqVJ*~@Y9I_>ooXa$%(X-)OYYufcn!5_?-Yz=zbq9vh|m0kz#j?_czV? zSlcOwj_rYKdR@T;NfCTd&r723uYRr>q%$1HyK=4_(cd{Dz5TINa z?B*Kq&^SDHw1C}*(Q1FP$A^3%J>>*`Bq)tF_%hI>*~Q%91*--aN~n7QoCtt`a6l>=|rtoG7I12xd-%>~|EvBd#1i@XZ+Qh?yf-?nCE zwl($IDk;j!DFTiJKoy@+_DN06j6Ir)pW<~GmK3w$V(B8v2afvzNO^z|2W7Mjgr~*H zagamD?DDYEB13k=(9O^%a{?AGcjYKhbgF->+vfhVOfo`%y8+5#0T?>j!=bluTid`M z6(d~N(6&p?i*b|a@>j}&rz`|QZ$L`oHt7l{>&z5-6!sKYH%`8UC^5aN-vsQSzZ4jw z_3QgUJcgp&p$_kCz}?$G;WyxsnO3dq)hRY(lO=zS0`EtYsZ z_eb-N7i$H9`OnoEpWV*+<(KAg)CrXjLA*_-CrACOg9X#l@a3;ff$kIZS~@|hQYFD^ zW2-9Ddt~GehATPKdnpr9bOMnr)BJFOwAU!YE!ioA}WN9?^gzpZLYAH7LGO9{*qeHP{HV&h2t zgL*N<=Tqa>DxZB{Ij5jF0olKy@bT&P!$?K;iZ)rn8H7~H&!$pe!2DUUoHL_pL zant0x)_K{zofjiZUe9Ad9c-02!*Kx?m7vXF`Z*$+UG~!3v8l5rPJ%_u{`#nlr%%93 zJ^uO6MIk&?y2Ja>?nGT5V}S)tL$_Fg<=p0wZeub6H9}xAJ#Tmj$pi2PFBe8VxELSN_&9 z>V(&O8IKPbIWn8*EOH>l&!6dCTLu)JS>z)$kTxO9!yjPuNXgx|VMTI-vpNKI)Ko7C zqw)FTk<^MyIZF;fBRFn>;5W$t_X5y!aF4PG&DAU5;~Njl0%ZWd##vFUJjgrKyyc4U zkABlFP*()0stixNJnu|*%b7CfcBENL!;gSmIe1PhV}P)pjqhOZ6PU%h(HJ&B$Ohqd zzCxy*?;wE*6Q_$a45k{f{&8R9Zq2+<6*sF2=jYP;4(d91A5-2?8_VnRG`>C}hKc9ZsJMB$e|0Am9r$z#?P(S3I{-fS@E$=}B zV&%s5djPTr^$8e#{kxfI=G=j;w*Ig0i18&8s16dMK^4T$Tb!q3_)^|~Gjkf@>1UCXTXU!46;j`s+_K04{>$lSs zxQjT~3sEFvvRKc#!4Nf|H#JIYs62WPxJ`DLa>@~`ze}311pfp@1I%QFrJ}SU&`4{D zzb zh9{q3i4}k{b-SD{RBkTt{%s9JJm<=ZtHKVOF{mN`e8~0vQqu7SnxE@bK%g*2{3#Qs zranfSAV`t?lFBW?IgF&MWjGukRzv2)F6`H7hwz2Ub?(UrO$7GyT_vW)&Hwgsz5#W7 z_~3BzWX^P%KL{F^_~ewi3fCb#j);%(uR2mjdp#3D0kw?bM0y0=r4WZDZgp7J-@ym6 zMjH1hY+TnU4Q0?@#pts zqxTqm8X@*&S-lnGH1RLanSK;+lu(giU^Pn~5!u3ttLW2VX^;z?myuMah4c49Bl2Kq zxIYc-RF(C82SMSx_a$a2gWUbS`*%nHG2y3TKj%oB6`)Pfag%!VUNm=~-6IPEH5wWj zv1)|-*!4941?>x38vuVE1S*ODmBqOB%YsrnSV?h(_HR`{F6tPjK?>$5c;x;rtERZ! z@1@LPmU@7Q^^<5(;d{S0Gc<=lM5OUsYtBUce0SAS=8H1Mig!AF{ppn5OTM4*Y;cw~ zs7$PE{Q;%(bfJ+FsEpA&CGuDKDyKG>*D9D?m2Vj-<2yCPnu5ut$I%CSR_e^WuB<^H z%gwBKce$hoaWz9T)+eb%{vh-s9 zL(Xtzg8uu-{+92q=#j+RWV(PLv=)wyM~vY~(7D-y1pAX`%O>B>!b8Apl6zn9D9nC6 z&#s`gvf4}`2{3m^wJ$O%9#O(Ko88bqYikt64Mbt3HS~Yg^^1947?;LIH&g?g2OVGU z?|du7D5NX8(KqUF#ayC3b;a!1w4`tDB|Mat^N`;I$vd?TAnUcfl2T#bzy8w_vW{;f zg503WPd%FCT++hn4$u2;uAV=t27qbfGB78P?#PzE5qznK#p*E~(gxK6`_S?S2|P=? zG`=rfJFxhY@{n)(UCznt01uhe8REMsj?ViHwdvZ#%>S5jkaqw`qieAo#jRxmk%76d zDOe`0Y(j&!=!G4(iKQ;n4A@_ZhDMG_lSaYuoW^=7|MbYT(?Lo5( zmbVW&5Aml*3N4sN85drILW&IK6DbV^k~(k?^{Z7Cgd&w+iuEW_7b&B>9Ajx4ONzV2ele>|Z_k7LEX4wlio?`;%_%TL$XYyL{9%v$0 zn>;afF?RB8Q1ld8n@Xgy)^(}81;UZJR+XME_57;Q(}jG^*$EB5U$@`E7jb|5nk9au z#iKE(Kkr51pzSJ3*XJrt;$$koIqe3QB{ft?twJi{L$j0i2aC89SHBLtHHBRkhp;Y2 zQYnhEn6C8qEO(SqD7(p)ye#X&LDOv`TomD_*;R#A7E5@{Uc@&^0p7Lp7=KGaaeoC+ldvnTV>Sb z+IbY{q_h)vZrDsZyY6!K(szMd8)XJ7Y))GJZqwX{5xzc5Y1~m4C$Zm{G?_262=30i zEaVwL9$yZFSMgqfZl)^J;jS{;tdYC7-aZ^?9H;6(<6~p~B1F6jU6$nb9G>2=wPiOy z<3|LTblMHTtxpK8>L&4u5nlb>-P?XPv04$2T!kL|=zU`1JeP|f=HjQB^sr}`jfkn8 z7lu@WiCyTe?$2;k<9`Q*BbY|NCt^GQGL|1|tY6n&8@tcAUYaGH)^0D`km6(<0{ZU$ zsM+DpJ=cNZt}Y#NIpnO*vgdl-uSuyY`Pm0>&>zxrX#&-?zVTuofmaz(3azmgx=cz9`p*S`1{ZI7p;VuH}8 zI}*)=1~Uyf-4(LzwtosOoJpeZ3uX5dW~BqraNcjxqDJl!X@PlkT%lF~O&s|u^lVyo z-<2>AFLz<iQ~g;u}>%U&{NPYNkB21(kvzc?t2C zfK6jCf$4AWItP5rFhO9rnVrqNHr@_--81evweW(RoB?_BC#*&{uoP@ux2N{F$D*)i zMz&+HQa;wm9bw1y6BLLi8Vq{U`x9dxy$>6I3z6 z19kO2j3qGuq1vO>iO7y&If8$&kK!sa3p0|?U#)_u~4%ZS|mT@*A+m^FJ8>N zoA0#9oC~icZeM#>Tw$p9^y>j&s36N(Nb4C(kUVkQG;KIKCb~QhyZ0}iZX2fA6JAE; z;9qQ?WJLc-?cQrGQJ8NBvYvU+vz$#)T{N3lPC^25F{yH~4Duc3hxW4hDsR~b4OCiuIj#z|SDq9u7a-|4pxKA`XKXxo z-^un^4Z@LNS@wn7@LD}wB}|@)ynH?h1B}dvcl%9;eZnZ8%NBPtr6j7Rn4@>yu$vn? z>m!OjFfu;?0nwWaPyspTtcLVv+ zV^5*}x@9I!QhSn1>%ACq9__r%icx#8={L{mitWZ1aM)UF$B}{l=SI~J-W{}nkk+gn zYTT809x=^zbn>)+F(u;4{b8Mbfx?xoi9*TT4qD6Q)?*TxL| zwCO4r3!~yl;`jtT0p!nXsRaUibu8ub-kbN9p|hb|ujHF>rfD0Wx4yk_PWC{TVE@9U zHao@|q!P27qF-0;iprx+(Yp~7w}yQ?oA1eb$WK_aTk-iSXb|t9Sfj$*f?QnjbGX7vokKl;t@AtKkvvEKA6ASHacEeqS2n9VT%xF2I z_y|$*v*%;+Y0pw11Q^dyjux0tVy&k&VyveHgVinkoygU)nTdF6Nvz}SXQamXuRlqf z>rAH2ucld%`r9&i#@5~alyp3k{X>LldgF?U47&_Qi$9BV5@p&Q8o}dZD?Q{7D)P|n z_(`G_Ri8}H+=?@w=dA^qam|WTl*_7-df+VA3yj1fSyu}aVJhzJN|tc{>_CyCy+Dtv zmcd2ecgcrp!;|W9&7kMnao{H+At0VHwV+^Ldkg{Hi2$KOaArr3bo^eFaavq&7dHL*8j&#nQB&F$JKQ_F+L&gRb0h^&I1jr-NiB4fEJUXyFaetY z+#W3uGk{9q>A8prY1Pk%`9`YF9}x1SL_gw9n$Qmr^(?G-2R%Csijmpy{AA{L3otgrYb&ZKK8ims&$q_`zz!@yM}i12Df@> zqyVTcs3KeK${4?^!}V;N@{3jKbzZ&|Y`Tc=RdjLT zAxbxiPpx$x7Wthv*0q)Z5VWAUoHy3e)!WQ4+U6+xEOm@YqzuKSn4i%^z>Esi*{`C= zMLi<1FB8OMzjTT9(uU+5TDqj{ssfM-rxxnHkEB)3s z{?9Ajr8fX80xM>kd`2b{K0KndOv_c2#j}8gcBL33wZl1Ws^8NHHB-G92SNctppqK$ z@a648>F*Uqqm!nt0A|Xf$@WxV*mG0#4zPrd|KSMYZ zE#T|UHV(&?vW!9P%0fg7LQd<=353)ywz-*mTagnzshKuFA7|DAot#qj`6_<7aKsg1 zzMK?=@9u41ns6RthK7dsOIAWm?egN%V*N+^fk2~OnM~LAdA1ttsY?)H9M;FTNepa1 zQBm8spBV$>>iR^<<(4>)2fhI=GN@ZjuId~~Ot?6uf3hg!XL&r40I~`|@6+2e_5|eU z_J>_z`Sl(X6wq~3G+}u{`L}5K&z=BU4K`>xbG0Ac_I7%}UC-jRESd-ap{lqiuKcSs zrM5)634MhUBnsEN)lCdD*K&TjkUrS!K3QKIm~@evsQGi1-p9}OSYjxrrsejNHrsXO zyJXqyB-B6(6(0NHP2I>e+nv3Jm}Qg02h_pkH3eh!YuD&8#hh=3dX1tuN3Cus3fQx- zeN~9Qct$@RwW4nKFzi<$uKzG$Ao^N6+Q65k@L(gw(f+P*9k^u8DKN)D$gL#~1QyC_ zriTT{!vm@&1uXhGKfNBn2R8-bVM;I%9-*o@VAOB@#M-@oSb5g`YuDE32ofkbdcWl( zBVc)$kellwJQ2>>tp*U~*rNHkWzM9-w*k|(zE`M>y{{G+lTF&ZQ5s|B0SXzAa8R{_ zXn%}el;yy|hIws*QJn+KvF?@R4>P>f#6Jk0RD`z!L?@11f?E!ezbA~rSu`-awEJ$O zM0IN9;ya5EKY9#}_KWk#Gc`H97*J=eb?vTs%nNrA@xPTn#_t z5}GspCi@#+%kW>kZ=gbZJCenfmCUj2kN*OEEXzZ;bp`wPW&ti`_sT(6X>4MO$k4QG zNM0hRe5;0J_IrCEV6#LBHLRo@roK~jl}z}U$rND!YKYLa3Phl-;c_F>b#Z0)y=A3< zh;6BPw$huojo0Z!t{P+Uo^N*x3)<=Hk6H`HYG2g#aRYYm2RGg60aT#D{&)c8$?z57 zKZCA%vgW}fkL3t3MM7KeJf_uh{Yd)a-O`EqQxe?HV7 zZ`*9`yTiUhPYNCezn=0pcOFboq_1Jx571lOWB}VO zDTusulfL7Eg8!n(9DCayBhjLx-LncGk$;dk37(2lFBY*xGawQ3P`^y1&A;~u_X9nk z@hu-I<<~gSs=c3)7pvQBp200X$eu^TXiQnpe+k7t@3FMyS+$~rP+Ggy+nOn`bN|FuLjv1a%cb7xj58#+jE z%y#Qqq5XRwklbFj_a+YSNnAA5)OA2BK_XkIwAx|-X9{r4Dqk&;*=Y+_Yh4 z9RO-B(3%mSH#ipn-n{MsP`m@eAxxkPrlIi%GrqCgxj3aXWaA1Gh`*v9#I34x``#T| z&k`BwMcYYZNT_y~ z!7BOp=iZ1D8~5 z0+AP@0^rNPFX`dirlN_Is)0SvUk z#oJl!bUrJ|g@PDtzi}o1QpG|+15y(6a?!#AGrP33#m!jD&CTE!ozo# zJ+mY6>T7Apml4M8wM$)B{aNbWp+E4c!b4xBk~Y-XbfZ<-#M$x2@r^S2tOEcTxQPkC zpz1#J^*JpyJgR^c=>cF&U2I9~-aXsKjz3e_qyY5v=*%G;4tU0*!KrtSs5BFN@up`E z4`VPg5dn)qt;K@WVT`K5+s?r<4$0r;ak^(T&)pe17)uVh0P0y~DmvJ%hM#^$OAkmt zd9;YpU+TUntw9d)8hhw3aFf?{n^Z~b8^p0>6_q{$i6Z6&&WJo4arLx{&-q$h8Nb_< z2_fb$0C0w!Mez~S%W7$^Ea21FBBRWMum=-9{*)g4z&%2AHz~0O@GYYN5LMc&|v5ma9kAi-6%&<&(h3cZ}3@7pW|h>v{=D(!A?h@ilS) zv&t9$0~3jm>y3go5vz(-Ggc)vs?zxY+?Ilg@NSa99NijrCQQyN&~s?LVvfFTTHUUn z5)1%9945T**h6LhQegha-Pni>YP~e&>=Z*U@-MXW66!i8X4}4F+ zTpe+7y;jDkg*?i}9|j;Zz`|Oyjt4dMiPk5_>Ao_v1LoFHuX!22)3Lo23SU-`PcWag z)yN-tRfCb6)h9hgy$Bl4F(~^P^b8#e?yVkjewtcPcpZxLC!kb z9!UldX4(~By)=WrToEQa6eMF7E+lRO)*U0Z)<<^(ATRYV`v0OZ-dh(h@EXH_Yr#;e z=Fm5VO-?Fc{kcsk73InOBdB&=r>p~g%p@WMEbC;Yg~VAWRorSSFs<$nWHin{1scnl zLwh3ke znW)EDfB*CU#7!Xo6{>|Eu&(^qt>yo67($IEnuAv_pn@@v*7RCCi<33!8^{M6(?Hi* zl+VlGgVIYKm7bvvR#2|^=f(epYr%iFH^{86!cJItiaSQg1sMbW4R25$4Q>DF->?Mw zVxi0(mVt}NEp8qZk68)zi3ODNgQ-t->bqioxt-y5eaGGtOAsEQo&fpdIg8yf_^iC> z#9SH-%~}5__B|OniU&=N`hMUWr%e7_TROiKCg-;M$}jK&A3B|DWA<#%y#9sw`-f4{ zsQ{`AnZ6MC+~uG5A;BVjGd{Oxz*JpxJdTYFC-}l+qBwoZhTKPe*6E)n!F{HG@^xK0Dh-gNP-ra#pG4iVt-a!+8G;NLzZ zczBx~rTG6244cNkhCKf-UflooulPUspx+TTs@*6!1Q)kE4Za-<2cRa$Ki)uOa=^@2 zuSSjbug%Bb`W0AV2LH=X1h(}5eEYwTbCB7Cx*Y`lv%uSRvEjW~GHG%&b)o^Cy zWC54)5%(6QBC6N*-FMOjR8p0(LbzuOh|{ALDXlWlQu?1uv+-cjcvu6P%C7j-)IR>bXmzeMD5pwc&wvJ~ax2nqZe}g`q>k9T)-^L&uq`_5IeJvL8bGP5c zE!=vUl0>!4*>=%-`sI=YJB8BY!W-3>6EeS15%Gx3ah)f7ZlU6793@y6>fwcXBeVM8 zZ{~hlc%(xgYmdk<_^(Yk*`V&7JL8m(=BGew&|;q#_+J5WEC(+$2gfT84&97(_5bz-dq*>C w3-AB=7nphtH^CPe|MM@nS=*btx*6L${m;MS|1*FGd + + + +Rancher Turtles is a [Rancher extension](../rancher-extensions.md) that provides integration between your Cluster API (CAPI) and Rancher to manage the lifecycle of provisioned Kubernetes clusters. The main functionality is outlined below: + +- Importing CAPI clusters into Rancher: Installing the Rancher Cluster Agent in CAPI provisioned clusters. +- Configuration support for the CAPI Operator. + +The [Overview](./overview.md) section outlines installation options, the main architecture, and a brief demo. For more detailed information, please view the [Rancher Turtles documentation](https://docs.rancher-turtles.com/docs/intro). diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md new file mode 100644 index 00000000000..4320dc87774 --- /dev/null +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -0,0 +1,210 @@ +--- +title: Overview +--- + + + + + +## Demo + + + +## Installing the Rancher Turtles Operator + +### Via Rancher Dashboard + +This is the recommended option for installing Rancher Turtles. + +Using the Rancher UI, and by adding the Turtles repository, we can let Rancher process the installation and configuration of the Cluster API Extension. + +:::caution +If you already have the Cluster API Operator installed in your cluster, you need to use the [manual helm installation method](./overview.md#via-helm-install) instead. +::: + +:::info prerequisite +Before installing Rancher Turtles in your Rancher environment, Rancher's `embedded-cluster-api` functionality must be disabled. This includes also cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. + +To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that applies these changes which are outlined below if manually removing: +- Disable the `embedded-cluster-api` feature in Rancher. +- Delete the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. +::: + +#### Installation + +- From your browser, access Rancher Manager and explore the **local** cluster. +- Using the left navigation panel, go to `Apps` -> `Repositories`. +- Click `Create` to add a new repository. +- Enter the following: + - **Name**: `turtles`. + - **Index URL**: https://rancher.github.io/turtles. +- Wait for the `turtles` repository to have a status of `Active`. +- Go to `Apps` -> `Charts`. +- Filter for `turtles`. +- Click `Rancher Turtles - the Cluster API Extension` +- Click `Install` -> `Next` -> `Install`. + +This will use the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or, alternatively, you can opt for the [manual installation via Helm](./overview.md#via-helm-install). If you are interested in learning more about the available values, you can check the Rancher Turtles [Helm chart reference guide](https://docs.rancher-turtles.com/docs/reference-guides/rancher-turtles-chart/values). + +The installation may take a few minutes and once completed you will see the following new deployments in the cluster: +- `rancher-turtles-system/rancher-turtles-controller-manager` +- `rancher-turtles-system/rancher-turtles-cluster-api-operator` +- `capi-system/capi-controller-manager` + +### Via Helm Install + +This section walks through different Helm installation options for Rancher Turtles depending on CAPI operator use: +- [Install Rancher Turtles with Cluster API Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency). +- [Install Rancher Turtles without Cluster API Operator](#install-rancher-turtles-without-cluster-api-operator-as-a-helm-dependency). + +The Cluster API Operator is required for installing Rancher Turtles and you can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. We recommend [installing as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) for the sake of simplicity, but the best option may depend on your specific configuration. + +CAPI Operator allows for handling the lifecycle of Cluster API providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). + +:::info prerequisite +Before installing Rancher Turtles in your Rancher environment, Rancher's `embedded-cluster-api` functionality must be disabled. This includes also cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. + +To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that applies these changes which are outlined below: +- Disable the `embedded-cluster-api` feature in Rancher. +- Delete the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. +::: + +#### Install Rancher Turtles with `Cluster API Operator` as a Helm dependency + +The `rancher-turtles` chart is available in https://rancher.github.io/turtles and this Helm repository must be added before proceeding with the installation: + +```bash +helm repo add turtles https://rancher.github.io/turtles +helm repo update +``` + +As mentioned before, installing Rancher Turtles requires the [Cluster API Operator](https://github.com/kubernetes-sigs/cluster-api-operator) and the Helm chart can handle its installation automatically with a minimum set of flags: + +```bash +helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ + -n rancher-turtles-system \ + --dependency-update \ + --create-namespace --wait \ + --timeout 180s +``` + +This operation could take a few minutes and once completed you can review the installed controllers listed below: +- `rancher-turtles-controller` +- `capi-operator` + +:::note +- If `cert-manager` is already available in the cluster, you can disable its installation as a Rancher Turtles dependency to avoid conflicts: +`--set cluster-api-operator.cert-manager.enabled=false` +- For a list of Rancher Turtles versions, refer to [Releases page](https://github.com/rancher/turtles/releases). +::: + +This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional Cluster API functionality. + +If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled. + +```bash +helm install ... + # Passing secret name and namespace for additional environment variables + --set cluster-api-operator.cluster-api.configSecret.name= +``` + +The following is an example of a user-managed secret `cluster-api-operator.cluster-api.configSecret.name=variables` with `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` feature flags set and an extra custom variable: + +```yaml title="secret.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: variables + namespace: rancher-turtles-system +type: Opaque +stringData: + CLUSTER_TOPOLOGY: "true" + EXP_CLUSTER_RESOURCE_SET: "true" + EXP_MACHINE_POOL: "true" + CUSTOM_ENV_VAR: "false" +``` + +:::info +For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://docs.rancher-turtles.com/docs/reference-guides/rancher-turtles-chart/values) +::: + +#### Install Rancher Turtles without `Cluster API Operator` as a Helm dependency + +:::note +Remember that if you opt for this installation option, you will need to manage the Cluster API Operator installation yourself. You can follow the [CAPI Operator guide](https://docs.rancher-turtles.com/docs/next/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. +::: + +The `rancher-turtles` chart is available in https://rancher.github.io/turtles and this Helm repository must be added before proceeding with the installation: + +```bash +helm repo add turtles https://rancher.github.io/turtles +helm repo update +``` + +and then it can be installed into the `rancher-turtles-system` namespace with: + +```bash +helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 + -n rancher-turtles-system + --set cluster-api-operator.enabled=false + --set cluster-api-operator.cluster-api.enabled=false + --create-namespace --wait + --dependency-update +``` + +As you can see, we are telling Helm to ignore installing `cluster-api-operator` as a dependency. + +This operation could take a few minutes and once completed you can review the installed controller listed below: +- `rancher-turtles-controller` + +## Uninstalling Rancher Turtles + +This section gives an overview of the Rancher Turtles uninstallation process. + +:::caution +When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the Cluster API Operator cleanup. This includes cleaning up Cluster API Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. + +To simplify uninstalling Rancher Turtles (via Rancher Manager or helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that applies these changes that are outlined below: +- Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. +- Deletes the CAPI `deployments` that are no longer needed. +::: + +To uninstall the Rancher Turtles Extension use the following helm command: + +```bash +helm uninstall -n rancher-turtles-system rancher-turtles --cascade foreground --wait +``` + +This may take a few minutes to complete. + +:::note +Remember that, if you use a different name for the installation or a different namespace, you may need to customize the command for your specific configuration. +::: + +Once uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: + +1. Create a `feature.yaml` file, with `embedded-cluster-api` set to true: +```yaml title="feature.yaml" +apiVersion: management.cattle.io/v3 +kind: Feature +metadata: + name: embedded-cluster-api +spec: + value: true +``` +2. Use `kubectl` to apply the `feature.yaml` file to the cluster: +```bash +kubectl apply -f feature.yaml +``` + +## Architecture Diagram + +Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the Cluster API (CAPI) for cluster management. + +![overview](/img/30000ft_view.png) + +## Security + +[SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. + +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, please visit the [Rancher Turtles Security](https://docs.rancher-turtles.com/docs/security/slsa) document. diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md new file mode 100644 index 00000000000..53ec6f4a66f --- /dev/null +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md @@ -0,0 +1,14 @@ +--- +title: Cluster API with Rancher Turtles +--- + + + + + +Rancher Turtles is a [Rancher extension](../rancher-extensions.md) that provides integration between your Cluster API (CAPI) and Rancher to manage the lifecycle of provisioned Kubernetes clusters. The main functionality is outlined below: + +- Importing CAPI clusters into Rancher: Installing the Rancher Cluster Agent in CAPI provisioned clusters. +- Configuration support for the CAPI Operator. + +The [Overview](./overview.md) section outlines installation options, the main architecture, and a brief demo. For more detailed information, please view the [Rancher Turtles documentation](https://docs.rancher-turtles.com/docs/intro). diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md new file mode 100644 index 00000000000..4320dc87774 --- /dev/null +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -0,0 +1,210 @@ +--- +title: Overview +--- + + + + + +## Demo + + + +## Installing the Rancher Turtles Operator + +### Via Rancher Dashboard + +This is the recommended option for installing Rancher Turtles. + +Using the Rancher UI, and by adding the Turtles repository, we can let Rancher process the installation and configuration of the Cluster API Extension. + +:::caution +If you already have the Cluster API Operator installed in your cluster, you need to use the [manual helm installation method](./overview.md#via-helm-install) instead. +::: + +:::info prerequisite +Before installing Rancher Turtles in your Rancher environment, Rancher's `embedded-cluster-api` functionality must be disabled. This includes also cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. + +To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that applies these changes which are outlined below if manually removing: +- Disable the `embedded-cluster-api` feature in Rancher. +- Delete the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. +::: + +#### Installation + +- From your browser, access Rancher Manager and explore the **local** cluster. +- Using the left navigation panel, go to `Apps` -> `Repositories`. +- Click `Create` to add a new repository. +- Enter the following: + - **Name**: `turtles`. + - **Index URL**: https://rancher.github.io/turtles. +- Wait for the `turtles` repository to have a status of `Active`. +- Go to `Apps` -> `Charts`. +- Filter for `turtles`. +- Click `Rancher Turtles - the Cluster API Extension` +- Click `Install` -> `Next` -> `Install`. + +This will use the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or, alternatively, you can opt for the [manual installation via Helm](./overview.md#via-helm-install). If you are interested in learning more about the available values, you can check the Rancher Turtles [Helm chart reference guide](https://docs.rancher-turtles.com/docs/reference-guides/rancher-turtles-chart/values). + +The installation may take a few minutes and once completed you will see the following new deployments in the cluster: +- `rancher-turtles-system/rancher-turtles-controller-manager` +- `rancher-turtles-system/rancher-turtles-cluster-api-operator` +- `capi-system/capi-controller-manager` + +### Via Helm Install + +This section walks through different Helm installation options for Rancher Turtles depending on CAPI operator use: +- [Install Rancher Turtles with Cluster API Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency). +- [Install Rancher Turtles without Cluster API Operator](#install-rancher-turtles-without-cluster-api-operator-as-a-helm-dependency). + +The Cluster API Operator is required for installing Rancher Turtles and you can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. We recommend [installing as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) for the sake of simplicity, but the best option may depend on your specific configuration. + +CAPI Operator allows for handling the lifecycle of Cluster API providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). + +:::info prerequisite +Before installing Rancher Turtles in your Rancher environment, Rancher's `embedded-cluster-api` functionality must be disabled. This includes also cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. + +To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that applies these changes which are outlined below: +- Disable the `embedded-cluster-api` feature in Rancher. +- Delete the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. +::: + +#### Install Rancher Turtles with `Cluster API Operator` as a Helm dependency + +The `rancher-turtles` chart is available in https://rancher.github.io/turtles and this Helm repository must be added before proceeding with the installation: + +```bash +helm repo add turtles https://rancher.github.io/turtles +helm repo update +``` + +As mentioned before, installing Rancher Turtles requires the [Cluster API Operator](https://github.com/kubernetes-sigs/cluster-api-operator) and the Helm chart can handle its installation automatically with a minimum set of flags: + +```bash +helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ + -n rancher-turtles-system \ + --dependency-update \ + --create-namespace --wait \ + --timeout 180s +``` + +This operation could take a few minutes and once completed you can review the installed controllers listed below: +- `rancher-turtles-controller` +- `capi-operator` + +:::note +- If `cert-manager` is already available in the cluster, you can disable its installation as a Rancher Turtles dependency to avoid conflicts: +`--set cluster-api-operator.cert-manager.enabled=false` +- For a list of Rancher Turtles versions, refer to [Releases page](https://github.com/rancher/turtles/releases). +::: + +This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional Cluster API functionality. + +If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled. + +```bash +helm install ... + # Passing secret name and namespace for additional environment variables + --set cluster-api-operator.cluster-api.configSecret.name= +``` + +The following is an example of a user-managed secret `cluster-api-operator.cluster-api.configSecret.name=variables` with `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` feature flags set and an extra custom variable: + +```yaml title="secret.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: variables + namespace: rancher-turtles-system +type: Opaque +stringData: + CLUSTER_TOPOLOGY: "true" + EXP_CLUSTER_RESOURCE_SET: "true" + EXP_MACHINE_POOL: "true" + CUSTOM_ENV_VAR: "false" +``` + +:::info +For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://docs.rancher-turtles.com/docs/reference-guides/rancher-turtles-chart/values) +::: + +#### Install Rancher Turtles without `Cluster API Operator` as a Helm dependency + +:::note +Remember that if you opt for this installation option, you will need to manage the Cluster API Operator installation yourself. You can follow the [CAPI Operator guide](https://docs.rancher-turtles.com/docs/next/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. +::: + +The `rancher-turtles` chart is available in https://rancher.github.io/turtles and this Helm repository must be added before proceeding with the installation: + +```bash +helm repo add turtles https://rancher.github.io/turtles +helm repo update +``` + +and then it can be installed into the `rancher-turtles-system` namespace with: + +```bash +helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 + -n rancher-turtles-system + --set cluster-api-operator.enabled=false + --set cluster-api-operator.cluster-api.enabled=false + --create-namespace --wait + --dependency-update +``` + +As you can see, we are telling Helm to ignore installing `cluster-api-operator` as a dependency. + +This operation could take a few minutes and once completed you can review the installed controller listed below: +- `rancher-turtles-controller` + +## Uninstalling Rancher Turtles + +This section gives an overview of the Rancher Turtles uninstallation process. + +:::caution +When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the Cluster API Operator cleanup. This includes cleaning up Cluster API Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. + +To simplify uninstalling Rancher Turtles (via Rancher Manager or helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that applies these changes that are outlined below: +- Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. +- Deletes the CAPI `deployments` that are no longer needed. +::: + +To uninstall the Rancher Turtles Extension use the following helm command: + +```bash +helm uninstall -n rancher-turtles-system rancher-turtles --cascade foreground --wait +``` + +This may take a few minutes to complete. + +:::note +Remember that, if you use a different name for the installation or a different namespace, you may need to customize the command for your specific configuration. +::: + +Once uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: + +1. Create a `feature.yaml` file, with `embedded-cluster-api` set to true: +```yaml title="feature.yaml" +apiVersion: management.cattle.io/v3 +kind: Feature +metadata: + name: embedded-cluster-api +spec: + value: true +``` +2. Use `kubectl` to apply the `feature.yaml` file to the cluster: +```bash +kubectl apply -f feature.yaml +``` + +## Architecture Diagram + +Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the Cluster API (CAPI) for cluster management. + +![overview](/img/30000ft_view.png) + +## Security + +[SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. + +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, please visit the [Rancher Turtles Security](https://docs.rancher-turtles.com/docs/security/slsa) document. diff --git a/versioned_sidebars/version-2.7-sidebars.json b/versioned_sidebars/version-2.7-sidebars.json index 7d677be558d..edb87c80c6f 100644 --- a/versioned_sidebars/version-2.7-sidebars.json +++ b/versioned_sidebars/version-2.7-sidebars.json @@ -1215,6 +1215,17 @@ ] }, "integrations-in-rancher/longhorn", + { + "type": "category", + "label": "Cluster API with Rancher Turtles", + "link": { + "type": "doc", + "id": "integrations-in-rancher/cluster-api/capi" + }, + "items": [ + "integrations-in-rancher/cluster-api/overview" + ] + }, { "type": "category", "label": "Logging", diff --git a/versioned_sidebars/version-2.8-sidebars.json b/versioned_sidebars/version-2.8-sidebars.json index 3e770dd537d..2dc5f196a5e 100644 --- a/versioned_sidebars/version-2.8-sidebars.json +++ b/versioned_sidebars/version-2.8-sidebars.json @@ -1175,6 +1175,17 @@ "integrations-in-rancher/cloud-marketplace/supportconfig" ] }, + { + "type": "category", + "label": "Cluster API with Rancher Turtles", + "link": { + "type": "doc", + "id": "integrations-in-rancher/cluster-api/capi" + }, + "items": [ + "integrations-in-rancher/cluster-api/overview" + ] + }, { "type": "category", "label": "CIS Scans", From a8470170cc7b10b949276e6087bc3bf030f56f6f Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Thu, 4 Apr 2024 09:04:08 -0700 Subject: [PATCH 02/67] Updated deprecated features table (#1214) * Add missing entries to deprecated features table * Remove reference to version in filename * Add redirects * Update canonical links * Update sidebars * Update header capitalization Co-authored-by: Marty Hernandez Avedon --------- Co-authored-by: Marty Hernandez Avedon --- .../faq/deprecated-features.md | 16 ++++----- docusaurus.config.js | 20 +++++++++++ sidebars.js | 2 +- ...ures-in-v2.5.md => deprecated-features.md} | 4 +-- .../version-2.6/faq/deprecated-features.md | 29 +++++++++------ .../faq/deprecated-features-in-v2.5.md | 30 ---------------- .../version-2.7/faq/deprecated-features.md | 35 +++++++++++++++++++ .../faq/deprecated-features.md} | 16 ++++----- versioned_sidebars/version-2.5-sidebars.json | 2 +- versioned_sidebars/version-2.6-sidebars.json | 2 +- versioned_sidebars/version-2.7-sidebars.json | 2 +- versioned_sidebars/version-2.8-sidebars.json | 2 +- 12 files changed, 92 insertions(+), 68 deletions(-) rename versioned_docs/version-2.8/faq/deprecated-features-in-v2.5.md => docs/faq/deprecated-features.md (61%) rename versioned_docs/version-2.5/faq/{deprecated-features-in-v2.5.md => deprecated-features.md} (92%) rename docs/faq/deprecated-features-in-v2.5.md => versioned_docs/version-2.6/faq/deprecated-features.md (61%) delete mode 100644 versioned_docs/version-2.7/faq/deprecated-features-in-v2.5.md create mode 100644 versioned_docs/version-2.7/faq/deprecated-features.md rename versioned_docs/{version-2.6/faq/deprecated-features-in-v2.5.md => version-2.8/faq/deprecated-features.md} (61%) diff --git a/versioned_docs/version-2.8/faq/deprecated-features-in-v2.5.md b/docs/faq/deprecated-features.md similarity index 61% rename from versioned_docs/version-2.8/faq/deprecated-features-in-v2.5.md rename to docs/faq/deprecated-features.md index 86313d48238..afda3bef11b 100644 --- a/versioned_docs/version-2.8/faq/deprecated-features-in-v2.5.md +++ b/docs/faq/deprecated-features.md @@ -3,10 +3,10 @@ title: Deprecated Features in Rancher --- - + -### What is Rancher's Deprecation policy? +### What is Rancher's deprecation policy? We have published our official deprecation policy in the support [terms of service](https://rancher.com/support-maintenance-terms). @@ -16,14 +16,10 @@ Rancher will publish deprecated features as part of the [release notes](https:// | Patch Version | Release Date | |---------------|---------------| -| [2.6.0](https://github.com/rancher/rancher/releases/tag/v2.6.0) | Aug 31, 2021 | -| [2.6.1](https://github.com/rancher/rancher/releases/tag/v2.6.1) | Oct 11, 2021 | -| [2.6.2](https://github.com/rancher/rancher/releases/tag/v2.6.2) | Oct 19, 2021 | -| [2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) | Dec 21, 2021 | -| [2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) | Mar 31, 2022 | -| [2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) | May 12, 2022 | -| [2.6.6](https://github.com/rancher/rancher/releases/tag/v2.6.6) | Jun 30, 2022 | - +| [2.8.3](https://github.com/rancher/rancher/releases/tag/v2.8.3) | Mar 28, 2024 | +| [2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2) | Feb 8, 2024 | +| [2.8.1](https://github.com/rancher/rancher/releases/tag/v2.8.1) | Jan 22, 2024 | +| [2.8.0](https://github.com/rancher/rancher/releases/tag/v2.8.0) | Dec 6, 2023 | ### What can I expect when a feature is marked for deprecation? diff --git a/docusaurus.config.js b/docusaurus.config.js index b75e5b9676f..03b865229a9 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -245,6 +245,26 @@ module.exports = { { fromExtensions: ['html', 'htm'], redirects: [ + { + to: '/faq/deprecated-features', + from: '/faq/deprecated-features-in-v2.5' + }, + { + to: '/v2.8/faq/deprecated-features', + from: '/v2.8/faq/deprecated-features-in-v2.5' + }, + { + to: '/v2.7/faq/deprecated-features', + from: '/v2.7/faq/deprecated-features-in-v2.5' + }, + { + to: '/v2.6/faq/deprecated-features', + from: '/v2.6/faq/deprecated-features-in-v2.5' + }, + { + to: '/v2.5/faq/deprecated-features', + from: '/v2.5/faq/deprecated-features-in-v2.5' + }, { // Redirects for pages-for-subheaders removal [2.0-2.4] to: '/v2.0-v2.4/how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-authentication', from: '/v2.0-v2.4/pages-for-subheaders/about-authentication' diff --git a/sidebars.js b/sidebars.js index 1ee600f77d4..2d62cda5e0e 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1306,7 +1306,7 @@ const sidebars = { label: 'FAQ', items: [ "faq/general-faq", - "faq/deprecated-features-in-v2.5", + "faq/deprecated-features", "faq/install-and-configure-kubectl", "faq/dockershim", "faq/technical-items", diff --git a/versioned_docs/version-2.5/faq/deprecated-features-in-v2.5.md b/versioned_docs/version-2.5/faq/deprecated-features.md similarity index 92% rename from versioned_docs/version-2.5/faq/deprecated-features-in-v2.5.md rename to versioned_docs/version-2.5/faq/deprecated-features.md index dba2fd3493a..80144c223e8 100644 --- a/versioned_docs/version-2.5/faq/deprecated-features-in-v2.5.md +++ b/versioned_docs/version-2.5/faq/deprecated-features.md @@ -3,10 +3,10 @@ title: Deprecated Features in Rancher v2.5 --- - + -### What is Rancher's Deprecation policy? +### What is Rancher's deprecation policy? Starting in Rancher 2.5 we have published our official deprecation policy in the support [terms of service](https://rancher.com/support-maintenance-terms). diff --git a/docs/faq/deprecated-features-in-v2.5.md b/versioned_docs/version-2.6/faq/deprecated-features.md similarity index 61% rename from docs/faq/deprecated-features-in-v2.5.md rename to versioned_docs/version-2.6/faq/deprecated-features.md index 86313d48238..a5ca0d9ebc5 100644 --- a/docs/faq/deprecated-features-in-v2.5.md +++ b/versioned_docs/version-2.6/faq/deprecated-features.md @@ -3,10 +3,10 @@ title: Deprecated Features in Rancher --- - + -### What is Rancher's Deprecation policy? +### What is Rancher's deprecation policy? We have published our official deprecation policy in the support [terms of service](https://rancher.com/support-maintenance-terms). @@ -14,16 +14,23 @@ We have published our official deprecation policy in the support [terms of servi Rancher will publish deprecated features as part of the [release notes](https://github.com/rancher/rancher/releases) for Rancher found on GitHub. Please consult the following patch releases for deprecated features: -| Patch Version | Release Date | +| Patch Version | Release Date | |---------------|---------------| -| [2.6.0](https://github.com/rancher/rancher/releases/tag/v2.6.0) | Aug 31, 2021 | -| [2.6.1](https://github.com/rancher/rancher/releases/tag/v2.6.1) | Oct 11, 2021 | -| [2.6.2](https://github.com/rancher/rancher/releases/tag/v2.6.2) | Oct 19, 2021 | -| [2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) | Dec 21, 2021 | -| [2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) | Mar 31, 2022 | -| [2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) | May 12, 2022 | -| [2.6.6](https://github.com/rancher/rancher/releases/tag/v2.6.6) | Jun 30, 2022 | - +| [2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | Feb 8, 2024 | +| [2.6.13](https://github.com/rancher/rancher/releases/tag/v2.6.13) | May 31, 2023 | +| [2.6.12](https://github.com/rancher/rancher/releases/tag/v2.6.12) | Apr 27, 2023 | +| [2.6.11](https://github.com/rancher/rancher/releases/tag/v2.6.11) | Mar 8, 2023 | +| [2.6.10](https://github.com/rancher/rancher/releases/tag/v2.6.10) | Jan 24, 2023 | +| [2.6.9](https://github.com/rancher/rancher/releases/tag/v2.6.9) | Oct 18, 2022 | +| [2.6.8](https://github.com/rancher/rancher/releases/tag/v2.6.8) | Aug 29, 2022 | +| [2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) | Aug 18, 2022 | +| [2.6.6](https://github.com/rancher/rancher/releases/tag/v2.6.6) | Jun 30, 2022 | +| [2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) | May 12, 2022 | +| [2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) | Mar 31, 2022 | +| [2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) | Dec 21, 2021 | +| [2.6.2](https://github.com/rancher/rancher/releases/tag/v2.6.2) | Oct 19, 2021 | +| [2.6.1](https://github.com/rancher/rancher/releases/tag/v2.6.1) | Oct 11, 2021 | +| [2.6.0](https://github.com/rancher/rancher/releases/tag/v2.6.0) | Aug 31, 2021 | ### What can I expect when a feature is marked for deprecation? diff --git a/versioned_docs/version-2.7/faq/deprecated-features-in-v2.5.md b/versioned_docs/version-2.7/faq/deprecated-features-in-v2.5.md deleted file mode 100644 index 86313d48238..00000000000 --- a/versioned_docs/version-2.7/faq/deprecated-features-in-v2.5.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -title: Deprecated Features in Rancher ---- - - - - - -### What is Rancher's Deprecation policy? - -We have published our official deprecation policy in the support [terms of service](https://rancher.com/support-maintenance-terms). - -### Where can I find out which features have been deprecated in Rancher? - -Rancher will publish deprecated features as part of the [release notes](https://github.com/rancher/rancher/releases) for Rancher found on GitHub. Please consult the following patch releases for deprecated features: - -| Patch Version | Release Date | -|---------------|---------------| -| [2.6.0](https://github.com/rancher/rancher/releases/tag/v2.6.0) | Aug 31, 2021 | -| [2.6.1](https://github.com/rancher/rancher/releases/tag/v2.6.1) | Oct 11, 2021 | -| [2.6.2](https://github.com/rancher/rancher/releases/tag/v2.6.2) | Oct 19, 2021 | -| [2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) | Dec 21, 2021 | -| [2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) | Mar 31, 2022 | -| [2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) | May 12, 2022 | -| [2.6.6](https://github.com/rancher/rancher/releases/tag/v2.6.6) | Jun 30, 2022 | - - -### What can I expect when a feature is marked for deprecation? - -In the release where functionality is marked as "Deprecated", it will still be available and supported allowing upgrades to follow the usual procedure. Once upgraded, users/admins should start planning to move away from the deprecated functionality before upgrading to the release it marked as removed. The recommendation for new deployments is to not use the deprecated feature. \ No newline at end of file diff --git a/versioned_docs/version-2.7/faq/deprecated-features.md b/versioned_docs/version-2.7/faq/deprecated-features.md new file mode 100644 index 00000000000..99ac433c6ca --- /dev/null +++ b/versioned_docs/version-2.7/faq/deprecated-features.md @@ -0,0 +1,35 @@ +--- +title: Deprecated Features in Rancher +--- + + + + + +### What is Rancher's deprecation policy? + +We have published our official deprecation policy in the support [terms of service](https://rancher.com/support-maintenance-terms). + +### Where can I find out which features have been deprecated in Rancher? + +Rancher will publish deprecated features as part of the [release notes](https://github.com/rancher/rancher/releases) for Rancher found on GitHub. Please consult the following patch releases for deprecated features: + +| Patch Version | Release Date | +|---------------|---------------| +| [2.7.12](https://github.com/rancher/rancher/releases/tag/v2.7.12) | Mar 28, 2024 | +| [2.7.11](https://github.com/rancher/rancher/releases/tag/v2.7.11) | Mar 1, 2024 | +| [2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) | Feb 8, 2024 | +| [2.7.9](https://github.com/rancher/rancher/releases/tag/v2.7.9) | Oct 26, 2023 | +| [2.7.8](https://github.com/rancher/rancher/releases/tag/v2.7.8) | Oct 5, 2023 | +| [2.7.7](https://github.com/rancher/rancher/releases/tag/v2.7.7) | Sep 28, 2023 | +| [2.7.6](https://github.com/rancher/rancher/releases/tag/v2.7.6) | Aug 30, 2023 | +| [2.7.5](https://github.com/rancher/rancher/releases/tag/v2.7.5) | Jun 29, 2023 | +| [2.7.4](https://github.com/rancher/rancher/releases/tag/v2.7.4) | May 31, 2023 | +| [2.7.3](https://github.com/rancher/rancher/releases/tag/v2.7.3) | Apr 24, 2023 | +| [2.7.2](https://github.com/rancher/rancher/releases/tag/v2.7.2) | Apr 11, 2023 | +| [2.7.1](https://github.com/rancher/rancher/releases/tag/v2.7.1) | Jan 24, 2023 | +| [2.7.0](https://github.com/rancher/rancher/releases/tag/v2.7.0) | Nov 16, 2022 | + +### What can I expect when a feature is marked for deprecation? + +In the release where functionality is marked as "Deprecated", it will still be available and supported allowing upgrades to follow the usual procedure. Once upgraded, users/admins should start planning to move away from the deprecated functionality before upgrading to the release it marked as removed. The recommendation for new deployments is to not use the deprecated feature. \ No newline at end of file diff --git a/versioned_docs/version-2.6/faq/deprecated-features-in-v2.5.md b/versioned_docs/version-2.8/faq/deprecated-features.md similarity index 61% rename from versioned_docs/version-2.6/faq/deprecated-features-in-v2.5.md rename to versioned_docs/version-2.8/faq/deprecated-features.md index 86313d48238..afda3bef11b 100644 --- a/versioned_docs/version-2.6/faq/deprecated-features-in-v2.5.md +++ b/versioned_docs/version-2.8/faq/deprecated-features.md @@ -3,10 +3,10 @@ title: Deprecated Features in Rancher --- - + -### What is Rancher's Deprecation policy? +### What is Rancher's deprecation policy? We have published our official deprecation policy in the support [terms of service](https://rancher.com/support-maintenance-terms). @@ -16,14 +16,10 @@ Rancher will publish deprecated features as part of the [release notes](https:// | Patch Version | Release Date | |---------------|---------------| -| [2.6.0](https://github.com/rancher/rancher/releases/tag/v2.6.0) | Aug 31, 2021 | -| [2.6.1](https://github.com/rancher/rancher/releases/tag/v2.6.1) | Oct 11, 2021 | -| [2.6.2](https://github.com/rancher/rancher/releases/tag/v2.6.2) | Oct 19, 2021 | -| [2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) | Dec 21, 2021 | -| [2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) | Mar 31, 2022 | -| [2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) | May 12, 2022 | -| [2.6.6](https://github.com/rancher/rancher/releases/tag/v2.6.6) | Jun 30, 2022 | - +| [2.8.3](https://github.com/rancher/rancher/releases/tag/v2.8.3) | Mar 28, 2024 | +| [2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2) | Feb 8, 2024 | +| [2.8.1](https://github.com/rancher/rancher/releases/tag/v2.8.1) | Jan 22, 2024 | +| [2.8.0](https://github.com/rancher/rancher/releases/tag/v2.8.0) | Dec 6, 2023 | ### What can I expect when a feature is marked for deprecation? diff --git a/versioned_sidebars/version-2.5-sidebars.json b/versioned_sidebars/version-2.5-sidebars.json index ef33985eeb8..f924143110e 100644 --- a/versioned_sidebars/version-2.5-sidebars.json +++ b/versioned_sidebars/version-2.5-sidebars.json @@ -1199,7 +1199,7 @@ "id": "faq" }, "items": [ - "faq/deprecated-features-in-v2.5", + "faq/deprecated-features", "faq/install-and-configure-kubectl", "faq/technical-items", "faq/security", diff --git a/versioned_sidebars/version-2.6-sidebars.json b/versioned_sidebars/version-2.6-sidebars.json index 851a06e7444..71c34c86d52 100644 --- a/versioned_sidebars/version-2.6-sidebars.json +++ b/versioned_sidebars/version-2.6-sidebars.json @@ -1193,7 +1193,7 @@ "label": "FAQ", "items": [ "faq/general-faq", - "faq/deprecated-features-in-v2.5", + "faq/deprecated-features", "faq/install-and-configure-kubectl", "faq/dockershim", "faq/technical-items", diff --git a/versioned_sidebars/version-2.7-sidebars.json b/versioned_sidebars/version-2.7-sidebars.json index 7d677be558d..4fc0bd58526 100644 --- a/versioned_sidebars/version-2.7-sidebars.json +++ b/versioned_sidebars/version-2.7-sidebars.json @@ -1266,7 +1266,7 @@ "label": "FAQ", "items": [ "faq/general-faq", - "faq/deprecated-features-in-v2.5", + "faq/deprecated-features", "faq/install-and-configure-kubectl", "faq/dockershim", "faq/technical-items", diff --git a/versioned_sidebars/version-2.8-sidebars.json b/versioned_sidebars/version-2.8-sidebars.json index 3e770dd537d..89b591673b2 100644 --- a/versioned_sidebars/version-2.8-sidebars.json +++ b/versioned_sidebars/version-2.8-sidebars.json @@ -1266,7 +1266,7 @@ "label": "FAQ", "items": [ "faq/general-faq", - "faq/deprecated-features-in-v2.5", + "faq/deprecated-features", "faq/install-and-configure-kubectl", "faq/dockershim", "faq/technical-items", From a4a0ab1903cb9f1b203801b519315ab6bd4ddc97 Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Thu, 4 Apr 2024 16:24:20 -0700 Subject: [PATCH 03/67] Remove 'deploy-apps-across-clusters' and subpages - Subpage on Fleet is duplicated content - Multi-cluster apps was previously deprecated and now removed --- .../deploy-apps-across-clusters.md | 21 -- .../deploy-apps-across-clusters/fleet.md | 71 ------- .../multi-cluster-apps.md | 179 ------------------ .../deploy-apps-across-clusters.md | 21 -- .../deploy-apps-across-clusters/fleet.md | 71 ------- .../multi-cluster-apps.md | 179 ------------------ .../deploy-apps-across-clusters.md | 21 -- .../deploy-apps-across-clusters/fleet.md | 71 ------- .../multi-cluster-apps.md | 179 ------------------ 9 files changed, 813 deletions(-) delete mode 100644 docs/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md delete mode 100644 docs/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md delete mode 100644 docs/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps.md delete mode 100644 versioned_docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md delete mode 100644 versioned_docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md delete mode 100644 versioned_docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps.md delete mode 100644 versioned_docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md delete mode 100644 versioned_docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md delete mode 100644 versioned_docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps.md diff --git a/docs/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md b/docs/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md deleted file mode 100644 index 7c3d85b867f..00000000000 --- a/docs/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: Deploying Applications across Clusters ---- - - - - - -Rancher offers several ways to deploy applications across clusters, depending on version. - -## Fleet - -Rancher v2.5 and later uses Fleet to deploy applications across clusters. - -Continuous Delivery with Fleet is GitOps at scale. For more information, refer to the [Fleet section](fleet.md). - -## Multi-cluster Apps - -In Rancher before v2.5, the multi-cluster apps feature was used to deploy applications across clusters. The multi-cluster apps feature is deprecated, but still available as a legacy feature. - -See the [multi-cluster app documentation](multi-cluster-apps.md) for more details. \ No newline at end of file diff --git a/docs/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md b/docs/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md deleted file mode 100644 index 2a47a9cdcfb..00000000000 --- a/docs/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -title: Continuous Delivery with Fleet ---- - - - - - -Continuous Delivery with Fleet is GitOps at scale. Fleet is designed to manage up to a million clusters. It's also lightweight enough that it works great for a [single cluster](https://fleet.rancher.io/installation#default-install) too, but it really shines when you get to a [large scale.](https://fleet.rancher.io/installation#configuration-for-multi-cluster) By large scale we mean either a lot of clusters, a lot of deployments, or a lot of teams in a single organization. - -Fleet is a separate project from Rancher, and can be installed on any Kubernetes cluster with Helm. - - -## Architecture - -For information about how Fleet works, see [this page.](../../../integrations-in-rancher/fleet/architecture.md) - -## Accessing Fleet in the Rancher UI - -Fleet comes preinstalled in Rancher and is managed by the **Continous Delivery** option in the Rancher UI. For additional information on Continuous Delivery and other Fleet troubleshooting tips, refer [here](https://fleet.rancher.io/troubleshooting). - -Users can leverage continuous delivery to deploy their applications to the Kubernetes clusters in the git repository without any manual operation by following **gitops** practice. - -Follow the steps below to access Continuous Delivery in the Rancher UI: - -1. Click **☰ > Continuous Delivery**. - -1. Select your namespace at the top of the menu, noting the following: - - By default,`fleet-default` is selected which includes all downstream clusters that are registered through Rancher. - - You may switch to `fleet-local`, which only contains the `local` cluster, or you may create your own workspace to which you may assign and move clusters. - - You can then manage clusters by clicking on **Clusters** on the left navigation bar. - -1. Click on **Gitrepos** on the left navigation bar to deploy the gitrepo into your clusters in the current workspace. - -1. Select your [git repository](https://fleet.rancher.io/gitrepo-add) and [target clusters/cluster group](https://fleet.rancher.io/gitrepo-targets). You can also create the cluster group in the UI by clicking on **Cluster Groups** from the left navigation bar. - -1. Once the gitrepo is deployed, you can monitor the application through the Rancher UI. - -## Windows Support - -For details on support for clusters with Windows nodes, see [this page.](../../../integrations-in-rancher/fleet/windows-support.md) - - -## GitHub Repository - -The Fleet Helm charts are available [here.](https://github.com/rancher/fleet/releases/latest) - - -## Using Fleet Behind a Proxy - -For details on using Fleet behind a proxy, see [this page.](../../../integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md) - -## Helm Chart Dependencies - -In order for Helm charts with dependencies to deploy successfully, you must run a manual command (as listed below), as it is up to the user to fulfill the dependency list. If you do not do this and proceed to clone your repository and run `helm install`, your installation will fail because the dependencies will be missing. - -The Helm chart in the git repository must include its dependencies in the charts subdirectory. You must either manually run `helm dependencies update $chart` or run `helm dependencies build $chart` locally, then commit the complete charts directory to your git repository. Note that you will update your commands with the applicable parameters. - -## Troubleshooting - ---- -* **Known Issue:** clientSecretName and helmSecretName secrets for Fleet gitrepos are not included in the backup nor restore created by the [backup-restore-operator](../backup-restore-and-disaster-recovery/back-up-rancher.md#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place. - -* **Temporary Workaround:**
-By default, user-defined secrets are not backed up in Fleet. It is necessary to recreate secrets if performing a disaster recovery restore or migration of Rancher into a fresh cluster. To modify resourceSet to include extra resources you want to backup, refer to docs [here](https://github.com/rancher/backup-restore-operator#user-flow). - ---- - -## Documentation - -The Fleet documentation is at [https://fleet.rancher.io/.](https://fleet.rancher.io/) diff --git a/docs/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps.md b/docs/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps.md deleted file mode 100644 index 61952f9dca3..00000000000 --- a/docs/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps.md +++ /dev/null @@ -1,179 +0,0 @@ ---- -title: Multi-cluster Apps ---- - - - - - -Typically, most applications are deployed on a single Kubernetes cluster, but there will be times you might want to deploy multiple copies of the same application across different clusters and/or projects. In Rancher, a _multi-cluster application_, is an application deployed using a Helm chart across multiple clusters. With the ability to deploy the same application across multiple clusters, it avoids the repetition of the same action on each cluster, which could introduce user error during application configuration. With multi-cluster applications, you can customize to have the same configuration across all projects/clusters as well as have the ability to change the configuration based on your target project. Since multi-cluster application is considered a single application, it's easy to manage and maintain this application. - -Any Helm charts from a global catalog can be used to deploy and manage multi-cluster applications. - -After creating a multi-cluster application, you can program a global DNS entry to make it easier to access the application. - -## Prerequisites - -### Permissions - -To create a multi-cluster app in Rancher, you must have at least one of the following permissions: - -- A [project-member role](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles) in the target cluster(s), which gives you the ability to create, read, update, and delete the workloads -- A [cluster owner role](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles) for the clusters(s) that include the target project(s) - -### Enable Legacy Features - -Because multi-cluster apps were deprecated and replaced with Fleet in Rancher v2.5, you will need to enable multi-cluster apps with a feature flag. - -1. In the upper left corner, click **☰ > Global Settings**. -1. Click **Feature Flags**. -1. Go to the `legacy` feature flag and click **Activate**. - -## Launching a Multi-Cluster App - -1. In the upper left corner, click **☰ > Multi-cluster Apps**. -1. Click **Launch**. -1. Find the application that you want to launch. -1. (Optional) Review the detailed descriptions, which are derived from the Helm chart's `README`. -1. Under **Configuration Options** enter a **Name** for the multi-cluster application. By default, this name is also used to create a Kubernetes namespace in each [target project](#targets) for the multi-cluster application. The namespace is named as `-`. -1. Select a **Template Version**. -1. Complete the [multi-cluster applications specific configuration options](#multi-cluster-app-configuration-options) as well as the [application configuration options](#application-configuration-options). -1. Select the **Members** who can [interact with the multi-cluster application](#members). -1. Add any [custom application configuration answers](#overriding-application-configuration-options-for-specific-projects) that would change the configuration for specific project(s) from the default application configuration answers. -1. Review the files in the **Preview** section. When you're satisfied, click **Launch**. - -**Result**: Your application is deployed to your chosen namespace. You can view the application status from the project's: - -## Multi-cluster App Configuration Options - -Rancher has divided the configuration option for the multi-cluster application into several sections. - -### Targets - -In the **Targets** section, select the projects that you want the application to be deployed in. The list of projects is based on what projects you have access to. For each project that you select, it will be added to the list, which shows the cluster name and project name that were selected. To remove a target project, click on **-**. - -### Upgrades - -In the **Upgrades** section, select the upgrade strategy to use, when you decide to upgrade your application. - -* **Rolling Update (batched):** When selecting this upgrade strategy, the number of applications upgraded at a time is based on the selected **Batch size** and the **Interval** specifies how many seconds to wait before starting the next batch of updates. - -* **Upgrade all apps simultaneously:** When selecting this upgrade strategy, all applications across all projects will be upgraded at the same time. - -### Roles - -In the **Roles** section, you define the role of the multi-cluster application. Typically, when a user [launches catalog applications](../helm-charts-in-rancher/helm-charts-in-rancher.md), that specific user's permissions are used for creation of all workloads/resources that is required by the app. - -For multi-cluster applications, the application is deployed by a _system user_ and is assigned as the creator of all underlying resources. A _system user_ is used instead of the actual user due to the fact that the actual user could be removed from one of the target projects. If the actual user was removed from one of the projects, then that user would no longer be able to manage the application for the other projects. - -Rancher will let you select from two options for Roles, **Project** and **Cluster**. Rancher will allow creation using any of these roles based on the user's permissions. - -- **Project** - This is the equivalent of a [project member](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles). If you select this role, Rancher will check that in all the target projects, the user has minimally the [project member](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles) role. While the user might not be explicitly granted the _project member_ role, if the user is an [administrator](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md), a [cluster owner](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles), or a [project owner](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles), then the user is considered to have the appropriate level of permissions. - -- **Cluster** - This is the equivalent of a [cluster owner](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles). If you select this role, Rancher will check that in all the target projects, the user has minimally the [cluster owner](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles) role. While the user might not be explicitly granted the _cluster owner_ role, if the user is an [administrator](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md), then the user is considered to have the appropriate level of permissions. - -When launching the application, Rancher will confirm if you have these permissions in the target projects before launching the application. - -:::note - -There are some applications like _Grafana_ or _Datadog_ that require access to specific cluster-scoped resources. These applications will require the _Cluster_ role. If you find out later that the application requires cluster roles, the multi-cluster application can be upgraded to update the roles. - -::: - -## Application Configuration Options - -For each Helm chart, there are a list of desired answers that must be entered in order to successfully deploy the chart. When entering answers, you must format them using the syntax rules found in [Using Helm: The format and limitations of –set](https://helm.sh/docs/intro/using_helm/#the-format-and-limitations-of---set), as Rancher passes them as `--set` flags to Helm. - -:::note Example - -When entering an answer that includes two values separated by a comma (i.e. `abc, bcd`), it is required to wrap the values with double quotes (i.e., ``"abc, bcd"``). - -::: - -### Using a questions.yml file - -If the Helm chart that you are deploying contains a `questions.yml` file, Rancher's UI will translate this file to display an easy to use UI to collect the answers for the questions. - -### Key Value Pairs for Native Helm Charts - -For native Helm charts (i.e., charts from the **Helm Stable** or **Helm Incubator** catalogs or a custom Helm chart repository, answers are provided as key value pairs in the **Answers** section. These answers are used to override the default values. - -### Members - -By default, multi-cluster applications can only be managed by the user who created it. In the **Members** section, other users can be added so that they can also help manage or view the multi-cluster application. - -1. Find the user that you want to add by typing in the member's name in the **Member** search box. - -2. Select the **Access Type** for that member. There are three access types for a multi-cluster project, but due to how the permissions of a multi-cluster application are launched, please read carefully to understand what these access types mean. - - - **Owner**: This access type can manage any configuration part of the multi-cluster application including the template version, the [multi-cluster applications specific configuration options](#Multi-cluster App Configuration Options), the [application specific configuration options](#application-configuration-options), the members who can interact with the multi-cluster application and the [custom application configuration answers](#overriding-application-configuration-options-for-specific-projects). Since a multi-cluster application is created with a different set of permissions from the user, any _owner_ of the multi-cluster application can manage/remove applications in [target projects](#targets) without explicitly having access to these project(s). Only trusted users should be provided with this access type. - - - **Member**: This access type can only modify the template version, the [application specific configuration options](#application-configuration-options) and the [custom application configuration answers](#overriding-application-configuration-options-for-specific-projects). Since a multi-cluster application is created with a different set of permissions from the user, any _member_ of the multi-cluster application can modify the application without explicitly having access to these project(s). Only trusted users should be provided with this access type. - - - **Read-only**: This access type cannot modify any configuration option for the multi-cluster application. Users can only view these applications. - - :::caution - - Please ensure only trusted users are given _Owner_ or _Member_ access as they will automatically be able to manage applications created for this multi-cluster application in target projects they might not have direct access to. - - ::: - -### Overriding Application Configuration Options for Specific Projects - -The ability to use the same configuration to deploy the same application across multiple clusters/projects is one of the main benefits of multi-cluster applications. There might be a specific project that requires a slightly different configuration option, but you want to manage that application with all the other matching applications. Instead of creating a brand new application, you can override specific [application specific configuration options](#application-configuration-options) for specific projects. - -1. In the **Answer Overrides** section, click **Add Override**. - -2. For each override, you can select the following: - - - **Scope**: Select which target projects you want to override the answer in the configuration option. - - - **Question**: Select which question you want to override. - - - **Answer**: Enter the answer that you want to be used instead. - -## Upgrading Multi-Cluster App Roles and Projects - -- **Changing Roles on an existing Multi-Cluster app** -The creator and any users added with the access-type "owner" to a multi-cluster app, can upgrade its Roles. When adding a new Role, we check if the user has that exact role in all current target projects. These checks allow the same relaxations for global admins, cluster owners and project-owners as described in the installation section for the field `Roles`. - -- **Adding/Removing target projects** -1. The creator and any users added with access-type "owner" to a multi-cluster app, can add or remove its target projects. When adding a new project, we check if the caller of this request has all Roles defined on multi-cluster app, in the new projects they want to add. The roles checks are again relaxed for global admins, cluster-owners and project-owners. -2. We do not do these membership checks when removing target projects. This is because the caller's permissions could have with respect to the target project, or the project could have been deleted and hence the caller wants to remove it from targets list. - - -## Multi-Cluster Application Management - -One of the benefits of using a multi-cluster application as opposed to multiple individual applications of the same type, is the ease of management. Multi-cluster applications can be cloned, upgraded or rolled back. - -:::note Prerequisite: - -The `legacy` feature flag needs to be enabled. - -::: - -1. In the upper left corner, click **☰ > Multi-cluster Apps**. - -2. Choose the multi-cluster application you want to take one of these actions on and click the **⋮**. Select one of the following options: - - * **Clone**: Creates another multi-cluster application with the same configuration. By using this option, you can easily duplicate a multi-cluster application. - * **Upgrade**: Upgrade your multi-cluster application to change some part of the configuration. When performing an upgrade for multi-cluster application, the [upgrade strategy](#upgrades) can be modified if you have the correct [access type](#members). - * **Rollback**: Rollback your application to a specific version. If after an upgrade, there are issues for your multi-cluster application for one or more of your [targets](#targets), Rancher has stored up to 10 versions of the multi-cluster application. Rolling back a multi-cluster application reverts the application for **all** target clusters and projects, not just the targets(s) affected by the upgrade issue. - -## Deleting a Multi-Cluster Application - -:::note Prerequisite: - -The `legacy` feature flag needs to be enabled. - -::: - -1. In the upper left corner, click **☰ > Multi-cluster Apps**. - -2. Choose the multi-cluster application you want to delete and click the **⋮ > Delete**. When deleting the multi-cluster application, all applications and namespaces are deleted in all of the target projects. - - :::note - - The applications in the target projects, that are created for a multi-cluster application, cannot be deleted individually. The applications can only be deleted when the multi-cluster application is deleted. - - ::: \ No newline at end of file diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md deleted file mode 100644 index 7c3d85b867f..00000000000 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: Deploying Applications across Clusters ---- - - - - - -Rancher offers several ways to deploy applications across clusters, depending on version. - -## Fleet - -Rancher v2.5 and later uses Fleet to deploy applications across clusters. - -Continuous Delivery with Fleet is GitOps at scale. For more information, refer to the [Fleet section](fleet.md). - -## Multi-cluster Apps - -In Rancher before v2.5, the multi-cluster apps feature was used to deploy applications across clusters. The multi-cluster apps feature is deprecated, but still available as a legacy feature. - -See the [multi-cluster app documentation](multi-cluster-apps.md) for more details. \ No newline at end of file diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md deleted file mode 100644 index e9021cb37ea..00000000000 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -title: Continuous Delivery with Fleet ---- - - - - - -Continuous Delivery with Fleet is GitOps at scale. Fleet is designed to manage up to a million clusters. It's also lightweight enough that it works great for a [single cluster](https://fleet.rancher.io/installation#default-install) too, but it really shines when you get to a [large scale.](https://fleet.rancher.io/installation#configuration-for-multi-cluster) By large scale we mean either a lot of clusters, a lot of deployments, or a lot of teams in a single organization. - -Fleet is a separate project from Rancher, and can be installed on any Kubernetes cluster with Helm. - - -## Architecture - -For information about how Fleet works, see [this page.](../../../integrations-in-rancher/fleet-gitops-at-scale/architecture.md) - -## Accessing Fleet in the Rancher UI - -Fleet comes preinstalled in Rancher and is managed by the **Continous Delivery** option in the Rancher UI. For additional information on Continuous Delivery and other Fleet troubleshooting tips, refer [here](https://fleet.rancher.io/troubleshooting). - -Users can leverage continuous delivery to deploy their applications to the Kubernetes clusters in the git repository without any manual operation by following **gitops** practice. - -Follow the steps below to access Continuous Delivery in the Rancher UI: - -1. Click **☰ > Continuous Delivery**. - -1. Select your namespace at the top of the menu, noting the following: - - By default,`fleet-default` is selected which includes all downstream clusters that are registered through Rancher. - - You may switch to `fleet-local`, which only contains the `local` cluster, or you may create your own workspace to which you may assign and move clusters. - - You can then manage clusters by clicking on **Clusters** on the left navigation bar. - -1. Click on **Gitrepos** on the left navigation bar to deploy the gitrepo into your clusters in the current workspace. - -1. Select your [git repository](https://fleet.rancher.io/gitrepo-add) and [target clusters/cluster group](https://fleet.rancher.io/gitrepo-targets). You can also create the cluster group in the UI by clicking on **Cluster Groups** from the left navigation bar. - -1. Once the gitrepo is deployed, you can monitor the application through the Rancher UI. - -## Windows Support - -For details on support for clusters with Windows nodes, see [this page.](../../../integrations-in-rancher/fleet-gitops-at-scale/windows-support.md) - - -## GitHub Repository - -The Fleet Helm charts are available [here.](https://github.com/rancher/fleet/releases/latest) - - -## Using Fleet Behind a Proxy - -For details on using Fleet behind a proxy, see [this page.](../../../integrations-in-rancher/fleet-gitops-at-scale/use-fleet-behind-a-proxy.md) - -## Helm Chart Dependencies - -In order for Helm charts with dependencies to deploy successfully, you must run a manual command (as listed below), as it is up to the user to fulfill the dependency list. If you do not do this and proceed to clone your repository and run `helm install`, your installation will fail because the dependencies will be missing. - -The Helm chart in the git repository must include its dependencies in the charts subdirectory. You must either manually run `helm dependencies update $chart` or run `helm dependencies build $chart` locally, then commit the complete charts directory to your git repository. Note that you will update your commands with the applicable parameters. - -## Troubleshooting - ---- -* **Known Issue:** clientSecretName and helmSecretName secrets for Fleet gitrepos are not included in the backup nor restore created by the [backup-restore-operator](../backup-restore-and-disaster-recovery/back-up-rancher.md#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place. - -* **Temporary Workaround:**
-By default, user-defined secrets are not backed up in Fleet. It is necessary to recreate secrets if performing a disaster recovery restore or migration of Rancher into a fresh cluster. To modify resourceSet to include extra resources you want to backup, refer to docs [here](https://github.com/rancher/backup-restore-operator#user-flow). - ---- - -## Documentation - -The Fleet documentation is at [https://fleet.rancher.io/.](https://fleet.rancher.io/) diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps.md deleted file mode 100644 index 61952f9dca3..00000000000 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps.md +++ /dev/null @@ -1,179 +0,0 @@ ---- -title: Multi-cluster Apps ---- - - - - - -Typically, most applications are deployed on a single Kubernetes cluster, but there will be times you might want to deploy multiple copies of the same application across different clusters and/or projects. In Rancher, a _multi-cluster application_, is an application deployed using a Helm chart across multiple clusters. With the ability to deploy the same application across multiple clusters, it avoids the repetition of the same action on each cluster, which could introduce user error during application configuration. With multi-cluster applications, you can customize to have the same configuration across all projects/clusters as well as have the ability to change the configuration based on your target project. Since multi-cluster application is considered a single application, it's easy to manage and maintain this application. - -Any Helm charts from a global catalog can be used to deploy and manage multi-cluster applications. - -After creating a multi-cluster application, you can program a global DNS entry to make it easier to access the application. - -## Prerequisites - -### Permissions - -To create a multi-cluster app in Rancher, you must have at least one of the following permissions: - -- A [project-member role](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles) in the target cluster(s), which gives you the ability to create, read, update, and delete the workloads -- A [cluster owner role](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles) for the clusters(s) that include the target project(s) - -### Enable Legacy Features - -Because multi-cluster apps were deprecated and replaced with Fleet in Rancher v2.5, you will need to enable multi-cluster apps with a feature flag. - -1. In the upper left corner, click **☰ > Global Settings**. -1. Click **Feature Flags**. -1. Go to the `legacy` feature flag and click **Activate**. - -## Launching a Multi-Cluster App - -1. In the upper left corner, click **☰ > Multi-cluster Apps**. -1. Click **Launch**. -1. Find the application that you want to launch. -1. (Optional) Review the detailed descriptions, which are derived from the Helm chart's `README`. -1. Under **Configuration Options** enter a **Name** for the multi-cluster application. By default, this name is also used to create a Kubernetes namespace in each [target project](#targets) for the multi-cluster application. The namespace is named as `-`. -1. Select a **Template Version**. -1. Complete the [multi-cluster applications specific configuration options](#multi-cluster-app-configuration-options) as well as the [application configuration options](#application-configuration-options). -1. Select the **Members** who can [interact with the multi-cluster application](#members). -1. Add any [custom application configuration answers](#overriding-application-configuration-options-for-specific-projects) that would change the configuration for specific project(s) from the default application configuration answers. -1. Review the files in the **Preview** section. When you're satisfied, click **Launch**. - -**Result**: Your application is deployed to your chosen namespace. You can view the application status from the project's: - -## Multi-cluster App Configuration Options - -Rancher has divided the configuration option for the multi-cluster application into several sections. - -### Targets - -In the **Targets** section, select the projects that you want the application to be deployed in. The list of projects is based on what projects you have access to. For each project that you select, it will be added to the list, which shows the cluster name and project name that were selected. To remove a target project, click on **-**. - -### Upgrades - -In the **Upgrades** section, select the upgrade strategy to use, when you decide to upgrade your application. - -* **Rolling Update (batched):** When selecting this upgrade strategy, the number of applications upgraded at a time is based on the selected **Batch size** and the **Interval** specifies how many seconds to wait before starting the next batch of updates. - -* **Upgrade all apps simultaneously:** When selecting this upgrade strategy, all applications across all projects will be upgraded at the same time. - -### Roles - -In the **Roles** section, you define the role of the multi-cluster application. Typically, when a user [launches catalog applications](../helm-charts-in-rancher/helm-charts-in-rancher.md), that specific user's permissions are used for creation of all workloads/resources that is required by the app. - -For multi-cluster applications, the application is deployed by a _system user_ and is assigned as the creator of all underlying resources. A _system user_ is used instead of the actual user due to the fact that the actual user could be removed from one of the target projects. If the actual user was removed from one of the projects, then that user would no longer be able to manage the application for the other projects. - -Rancher will let you select from two options for Roles, **Project** and **Cluster**. Rancher will allow creation using any of these roles based on the user's permissions. - -- **Project** - This is the equivalent of a [project member](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles). If you select this role, Rancher will check that in all the target projects, the user has minimally the [project member](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles) role. While the user might not be explicitly granted the _project member_ role, if the user is an [administrator](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md), a [cluster owner](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles), or a [project owner](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles), then the user is considered to have the appropriate level of permissions. - -- **Cluster** - This is the equivalent of a [cluster owner](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles). If you select this role, Rancher will check that in all the target projects, the user has minimally the [cluster owner](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles) role. While the user might not be explicitly granted the _cluster owner_ role, if the user is an [administrator](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md), then the user is considered to have the appropriate level of permissions. - -When launching the application, Rancher will confirm if you have these permissions in the target projects before launching the application. - -:::note - -There are some applications like _Grafana_ or _Datadog_ that require access to specific cluster-scoped resources. These applications will require the _Cluster_ role. If you find out later that the application requires cluster roles, the multi-cluster application can be upgraded to update the roles. - -::: - -## Application Configuration Options - -For each Helm chart, there are a list of desired answers that must be entered in order to successfully deploy the chart. When entering answers, you must format them using the syntax rules found in [Using Helm: The format and limitations of –set](https://helm.sh/docs/intro/using_helm/#the-format-and-limitations-of---set), as Rancher passes them as `--set` flags to Helm. - -:::note Example - -When entering an answer that includes two values separated by a comma (i.e. `abc, bcd`), it is required to wrap the values with double quotes (i.e., ``"abc, bcd"``). - -::: - -### Using a questions.yml file - -If the Helm chart that you are deploying contains a `questions.yml` file, Rancher's UI will translate this file to display an easy to use UI to collect the answers for the questions. - -### Key Value Pairs for Native Helm Charts - -For native Helm charts (i.e., charts from the **Helm Stable** or **Helm Incubator** catalogs or a custom Helm chart repository, answers are provided as key value pairs in the **Answers** section. These answers are used to override the default values. - -### Members - -By default, multi-cluster applications can only be managed by the user who created it. In the **Members** section, other users can be added so that they can also help manage or view the multi-cluster application. - -1. Find the user that you want to add by typing in the member's name in the **Member** search box. - -2. Select the **Access Type** for that member. There are three access types for a multi-cluster project, but due to how the permissions of a multi-cluster application are launched, please read carefully to understand what these access types mean. - - - **Owner**: This access type can manage any configuration part of the multi-cluster application including the template version, the [multi-cluster applications specific configuration options](#Multi-cluster App Configuration Options), the [application specific configuration options](#application-configuration-options), the members who can interact with the multi-cluster application and the [custom application configuration answers](#overriding-application-configuration-options-for-specific-projects). Since a multi-cluster application is created with a different set of permissions from the user, any _owner_ of the multi-cluster application can manage/remove applications in [target projects](#targets) without explicitly having access to these project(s). Only trusted users should be provided with this access type. - - - **Member**: This access type can only modify the template version, the [application specific configuration options](#application-configuration-options) and the [custom application configuration answers](#overriding-application-configuration-options-for-specific-projects). Since a multi-cluster application is created with a different set of permissions from the user, any _member_ of the multi-cluster application can modify the application without explicitly having access to these project(s). Only trusted users should be provided with this access type. - - - **Read-only**: This access type cannot modify any configuration option for the multi-cluster application. Users can only view these applications. - - :::caution - - Please ensure only trusted users are given _Owner_ or _Member_ access as they will automatically be able to manage applications created for this multi-cluster application in target projects they might not have direct access to. - - ::: - -### Overriding Application Configuration Options for Specific Projects - -The ability to use the same configuration to deploy the same application across multiple clusters/projects is one of the main benefits of multi-cluster applications. There might be a specific project that requires a slightly different configuration option, but you want to manage that application with all the other matching applications. Instead of creating a brand new application, you can override specific [application specific configuration options](#application-configuration-options) for specific projects. - -1. In the **Answer Overrides** section, click **Add Override**. - -2. For each override, you can select the following: - - - **Scope**: Select which target projects you want to override the answer in the configuration option. - - - **Question**: Select which question you want to override. - - - **Answer**: Enter the answer that you want to be used instead. - -## Upgrading Multi-Cluster App Roles and Projects - -- **Changing Roles on an existing Multi-Cluster app** -The creator and any users added with the access-type "owner" to a multi-cluster app, can upgrade its Roles. When adding a new Role, we check if the user has that exact role in all current target projects. These checks allow the same relaxations for global admins, cluster owners and project-owners as described in the installation section for the field `Roles`. - -- **Adding/Removing target projects** -1. The creator and any users added with access-type "owner" to a multi-cluster app, can add or remove its target projects. When adding a new project, we check if the caller of this request has all Roles defined on multi-cluster app, in the new projects they want to add. The roles checks are again relaxed for global admins, cluster-owners and project-owners. -2. We do not do these membership checks when removing target projects. This is because the caller's permissions could have with respect to the target project, or the project could have been deleted and hence the caller wants to remove it from targets list. - - -## Multi-Cluster Application Management - -One of the benefits of using a multi-cluster application as opposed to multiple individual applications of the same type, is the ease of management. Multi-cluster applications can be cloned, upgraded or rolled back. - -:::note Prerequisite: - -The `legacy` feature flag needs to be enabled. - -::: - -1. In the upper left corner, click **☰ > Multi-cluster Apps**. - -2. Choose the multi-cluster application you want to take one of these actions on and click the **⋮**. Select one of the following options: - - * **Clone**: Creates another multi-cluster application with the same configuration. By using this option, you can easily duplicate a multi-cluster application. - * **Upgrade**: Upgrade your multi-cluster application to change some part of the configuration. When performing an upgrade for multi-cluster application, the [upgrade strategy](#upgrades) can be modified if you have the correct [access type](#members). - * **Rollback**: Rollback your application to a specific version. If after an upgrade, there are issues for your multi-cluster application for one or more of your [targets](#targets), Rancher has stored up to 10 versions of the multi-cluster application. Rolling back a multi-cluster application reverts the application for **all** target clusters and projects, not just the targets(s) affected by the upgrade issue. - -## Deleting a Multi-Cluster Application - -:::note Prerequisite: - -The `legacy` feature flag needs to be enabled. - -::: - -1. In the upper left corner, click **☰ > Multi-cluster Apps**. - -2. Choose the multi-cluster application you want to delete and click the **⋮ > Delete**. When deleting the multi-cluster application, all applications and namespaces are deleted in all of the target projects. - - :::note - - The applications in the target projects, that are created for a multi-cluster application, cannot be deleted individually. The applications can only be deleted when the multi-cluster application is deleted. - - ::: \ No newline at end of file diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md deleted file mode 100644 index 7c3d85b867f..00000000000 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: Deploying Applications across Clusters ---- - - - - - -Rancher offers several ways to deploy applications across clusters, depending on version. - -## Fleet - -Rancher v2.5 and later uses Fleet to deploy applications across clusters. - -Continuous Delivery with Fleet is GitOps at scale. For more information, refer to the [Fleet section](fleet.md). - -## Multi-cluster Apps - -In Rancher before v2.5, the multi-cluster apps feature was used to deploy applications across clusters. The multi-cluster apps feature is deprecated, but still available as a legacy feature. - -See the [multi-cluster app documentation](multi-cluster-apps.md) for more details. \ No newline at end of file diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md deleted file mode 100644 index 2a47a9cdcfb..00000000000 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md +++ /dev/null @@ -1,71 +0,0 @@ ---- -title: Continuous Delivery with Fleet ---- - - - - - -Continuous Delivery with Fleet is GitOps at scale. Fleet is designed to manage up to a million clusters. It's also lightweight enough that it works great for a [single cluster](https://fleet.rancher.io/installation#default-install) too, but it really shines when you get to a [large scale.](https://fleet.rancher.io/installation#configuration-for-multi-cluster) By large scale we mean either a lot of clusters, a lot of deployments, or a lot of teams in a single organization. - -Fleet is a separate project from Rancher, and can be installed on any Kubernetes cluster with Helm. - - -## Architecture - -For information about how Fleet works, see [this page.](../../../integrations-in-rancher/fleet/architecture.md) - -## Accessing Fleet in the Rancher UI - -Fleet comes preinstalled in Rancher and is managed by the **Continous Delivery** option in the Rancher UI. For additional information on Continuous Delivery and other Fleet troubleshooting tips, refer [here](https://fleet.rancher.io/troubleshooting). - -Users can leverage continuous delivery to deploy their applications to the Kubernetes clusters in the git repository without any manual operation by following **gitops** practice. - -Follow the steps below to access Continuous Delivery in the Rancher UI: - -1. Click **☰ > Continuous Delivery**. - -1. Select your namespace at the top of the menu, noting the following: - - By default,`fleet-default` is selected which includes all downstream clusters that are registered through Rancher. - - You may switch to `fleet-local`, which only contains the `local` cluster, or you may create your own workspace to which you may assign and move clusters. - - You can then manage clusters by clicking on **Clusters** on the left navigation bar. - -1. Click on **Gitrepos** on the left navigation bar to deploy the gitrepo into your clusters in the current workspace. - -1. Select your [git repository](https://fleet.rancher.io/gitrepo-add) and [target clusters/cluster group](https://fleet.rancher.io/gitrepo-targets). You can also create the cluster group in the UI by clicking on **Cluster Groups** from the left navigation bar. - -1. Once the gitrepo is deployed, you can monitor the application through the Rancher UI. - -## Windows Support - -For details on support for clusters with Windows nodes, see [this page.](../../../integrations-in-rancher/fleet/windows-support.md) - - -## GitHub Repository - -The Fleet Helm charts are available [here.](https://github.com/rancher/fleet/releases/latest) - - -## Using Fleet Behind a Proxy - -For details on using Fleet behind a proxy, see [this page.](../../../integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md) - -## Helm Chart Dependencies - -In order for Helm charts with dependencies to deploy successfully, you must run a manual command (as listed below), as it is up to the user to fulfill the dependency list. If you do not do this and proceed to clone your repository and run `helm install`, your installation will fail because the dependencies will be missing. - -The Helm chart in the git repository must include its dependencies in the charts subdirectory. You must either manually run `helm dependencies update $chart` or run `helm dependencies build $chart` locally, then commit the complete charts directory to your git repository. Note that you will update your commands with the applicable parameters. - -## Troubleshooting - ---- -* **Known Issue:** clientSecretName and helmSecretName secrets for Fleet gitrepos are not included in the backup nor restore created by the [backup-restore-operator](../backup-restore-and-disaster-recovery/back-up-rancher.md#1-install-the-rancher-backup-operator). We will update the community once a permanent solution is in place. - -* **Temporary Workaround:**
-By default, user-defined secrets are not backed up in Fleet. It is necessary to recreate secrets if performing a disaster recovery restore or migration of Rancher into a fresh cluster. To modify resourceSet to include extra resources you want to backup, refer to docs [here](https://github.com/rancher/backup-restore-operator#user-flow). - ---- - -## Documentation - -The Fleet documentation is at [https://fleet.rancher.io/.](https://fleet.rancher.io/) diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps.md deleted file mode 100644 index 61952f9dca3..00000000000 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps.md +++ /dev/null @@ -1,179 +0,0 @@ ---- -title: Multi-cluster Apps ---- - - - - - -Typically, most applications are deployed on a single Kubernetes cluster, but there will be times you might want to deploy multiple copies of the same application across different clusters and/or projects. In Rancher, a _multi-cluster application_, is an application deployed using a Helm chart across multiple clusters. With the ability to deploy the same application across multiple clusters, it avoids the repetition of the same action on each cluster, which could introduce user error during application configuration. With multi-cluster applications, you can customize to have the same configuration across all projects/clusters as well as have the ability to change the configuration based on your target project. Since multi-cluster application is considered a single application, it's easy to manage and maintain this application. - -Any Helm charts from a global catalog can be used to deploy and manage multi-cluster applications. - -After creating a multi-cluster application, you can program a global DNS entry to make it easier to access the application. - -## Prerequisites - -### Permissions - -To create a multi-cluster app in Rancher, you must have at least one of the following permissions: - -- A [project-member role](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles) in the target cluster(s), which gives you the ability to create, read, update, and delete the workloads -- A [cluster owner role](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles) for the clusters(s) that include the target project(s) - -### Enable Legacy Features - -Because multi-cluster apps were deprecated and replaced with Fleet in Rancher v2.5, you will need to enable multi-cluster apps with a feature flag. - -1. In the upper left corner, click **☰ > Global Settings**. -1. Click **Feature Flags**. -1. Go to the `legacy` feature flag and click **Activate**. - -## Launching a Multi-Cluster App - -1. In the upper left corner, click **☰ > Multi-cluster Apps**. -1. Click **Launch**. -1. Find the application that you want to launch. -1. (Optional) Review the detailed descriptions, which are derived from the Helm chart's `README`. -1. Under **Configuration Options** enter a **Name** for the multi-cluster application. By default, this name is also used to create a Kubernetes namespace in each [target project](#targets) for the multi-cluster application. The namespace is named as `-`. -1. Select a **Template Version**. -1. Complete the [multi-cluster applications specific configuration options](#multi-cluster-app-configuration-options) as well as the [application configuration options](#application-configuration-options). -1. Select the **Members** who can [interact with the multi-cluster application](#members). -1. Add any [custom application configuration answers](#overriding-application-configuration-options-for-specific-projects) that would change the configuration for specific project(s) from the default application configuration answers. -1. Review the files in the **Preview** section. When you're satisfied, click **Launch**. - -**Result**: Your application is deployed to your chosen namespace. You can view the application status from the project's: - -## Multi-cluster App Configuration Options - -Rancher has divided the configuration option for the multi-cluster application into several sections. - -### Targets - -In the **Targets** section, select the projects that you want the application to be deployed in. The list of projects is based on what projects you have access to. For each project that you select, it will be added to the list, which shows the cluster name and project name that were selected. To remove a target project, click on **-**. - -### Upgrades - -In the **Upgrades** section, select the upgrade strategy to use, when you decide to upgrade your application. - -* **Rolling Update (batched):** When selecting this upgrade strategy, the number of applications upgraded at a time is based on the selected **Batch size** and the **Interval** specifies how many seconds to wait before starting the next batch of updates. - -* **Upgrade all apps simultaneously:** When selecting this upgrade strategy, all applications across all projects will be upgraded at the same time. - -### Roles - -In the **Roles** section, you define the role of the multi-cluster application. Typically, when a user [launches catalog applications](../helm-charts-in-rancher/helm-charts-in-rancher.md), that specific user's permissions are used for creation of all workloads/resources that is required by the app. - -For multi-cluster applications, the application is deployed by a _system user_ and is assigned as the creator of all underlying resources. A _system user_ is used instead of the actual user due to the fact that the actual user could be removed from one of the target projects. If the actual user was removed from one of the projects, then that user would no longer be able to manage the application for the other projects. - -Rancher will let you select from two options for Roles, **Project** and **Cluster**. Rancher will allow creation using any of these roles based on the user's permissions. - -- **Project** - This is the equivalent of a [project member](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles). If you select this role, Rancher will check that in all the target projects, the user has minimally the [project member](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles) role. While the user might not be explicitly granted the _project member_ role, if the user is an [administrator](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md), a [cluster owner](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles), or a [project owner](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles), then the user is considered to have the appropriate level of permissions. - -- **Cluster** - This is the equivalent of a [cluster owner](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles). If you select this role, Rancher will check that in all the target projects, the user has minimally the [cluster owner](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#project-roles) role. While the user might not be explicitly granted the _cluster owner_ role, if the user is an [administrator](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md), then the user is considered to have the appropriate level of permissions. - -When launching the application, Rancher will confirm if you have these permissions in the target projects before launching the application. - -:::note - -There are some applications like _Grafana_ or _Datadog_ that require access to specific cluster-scoped resources. These applications will require the _Cluster_ role. If you find out later that the application requires cluster roles, the multi-cluster application can be upgraded to update the roles. - -::: - -## Application Configuration Options - -For each Helm chart, there are a list of desired answers that must be entered in order to successfully deploy the chart. When entering answers, you must format them using the syntax rules found in [Using Helm: The format and limitations of –set](https://helm.sh/docs/intro/using_helm/#the-format-and-limitations-of---set), as Rancher passes them as `--set` flags to Helm. - -:::note Example - -When entering an answer that includes two values separated by a comma (i.e. `abc, bcd`), it is required to wrap the values with double quotes (i.e., ``"abc, bcd"``). - -::: - -### Using a questions.yml file - -If the Helm chart that you are deploying contains a `questions.yml` file, Rancher's UI will translate this file to display an easy to use UI to collect the answers for the questions. - -### Key Value Pairs for Native Helm Charts - -For native Helm charts (i.e., charts from the **Helm Stable** or **Helm Incubator** catalogs or a custom Helm chart repository, answers are provided as key value pairs in the **Answers** section. These answers are used to override the default values. - -### Members - -By default, multi-cluster applications can only be managed by the user who created it. In the **Members** section, other users can be added so that they can also help manage or view the multi-cluster application. - -1. Find the user that you want to add by typing in the member's name in the **Member** search box. - -2. Select the **Access Type** for that member. There are three access types for a multi-cluster project, but due to how the permissions of a multi-cluster application are launched, please read carefully to understand what these access types mean. - - - **Owner**: This access type can manage any configuration part of the multi-cluster application including the template version, the [multi-cluster applications specific configuration options](#Multi-cluster App Configuration Options), the [application specific configuration options](#application-configuration-options), the members who can interact with the multi-cluster application and the [custom application configuration answers](#overriding-application-configuration-options-for-specific-projects). Since a multi-cluster application is created with a different set of permissions from the user, any _owner_ of the multi-cluster application can manage/remove applications in [target projects](#targets) without explicitly having access to these project(s). Only trusted users should be provided with this access type. - - - **Member**: This access type can only modify the template version, the [application specific configuration options](#application-configuration-options) and the [custom application configuration answers](#overriding-application-configuration-options-for-specific-projects). Since a multi-cluster application is created with a different set of permissions from the user, any _member_ of the multi-cluster application can modify the application without explicitly having access to these project(s). Only trusted users should be provided with this access type. - - - **Read-only**: This access type cannot modify any configuration option for the multi-cluster application. Users can only view these applications. - - :::caution - - Please ensure only trusted users are given _Owner_ or _Member_ access as they will automatically be able to manage applications created for this multi-cluster application in target projects they might not have direct access to. - - ::: - -### Overriding Application Configuration Options for Specific Projects - -The ability to use the same configuration to deploy the same application across multiple clusters/projects is one of the main benefits of multi-cluster applications. There might be a specific project that requires a slightly different configuration option, but you want to manage that application with all the other matching applications. Instead of creating a brand new application, you can override specific [application specific configuration options](#application-configuration-options) for specific projects. - -1. In the **Answer Overrides** section, click **Add Override**. - -2. For each override, you can select the following: - - - **Scope**: Select which target projects you want to override the answer in the configuration option. - - - **Question**: Select which question you want to override. - - - **Answer**: Enter the answer that you want to be used instead. - -## Upgrading Multi-Cluster App Roles and Projects - -- **Changing Roles on an existing Multi-Cluster app** -The creator and any users added with the access-type "owner" to a multi-cluster app, can upgrade its Roles. When adding a new Role, we check if the user has that exact role in all current target projects. These checks allow the same relaxations for global admins, cluster owners and project-owners as described in the installation section for the field `Roles`. - -- **Adding/Removing target projects** -1. The creator and any users added with access-type "owner" to a multi-cluster app, can add or remove its target projects. When adding a new project, we check if the caller of this request has all Roles defined on multi-cluster app, in the new projects they want to add. The roles checks are again relaxed for global admins, cluster-owners and project-owners. -2. We do not do these membership checks when removing target projects. This is because the caller's permissions could have with respect to the target project, or the project could have been deleted and hence the caller wants to remove it from targets list. - - -## Multi-Cluster Application Management - -One of the benefits of using a multi-cluster application as opposed to multiple individual applications of the same type, is the ease of management. Multi-cluster applications can be cloned, upgraded or rolled back. - -:::note Prerequisite: - -The `legacy` feature flag needs to be enabled. - -::: - -1. In the upper left corner, click **☰ > Multi-cluster Apps**. - -2. Choose the multi-cluster application you want to take one of these actions on and click the **⋮**. Select one of the following options: - - * **Clone**: Creates another multi-cluster application with the same configuration. By using this option, you can easily duplicate a multi-cluster application. - * **Upgrade**: Upgrade your multi-cluster application to change some part of the configuration. When performing an upgrade for multi-cluster application, the [upgrade strategy](#upgrades) can be modified if you have the correct [access type](#members). - * **Rollback**: Rollback your application to a specific version. If after an upgrade, there are issues for your multi-cluster application for one or more of your [targets](#targets), Rancher has stored up to 10 versions of the multi-cluster application. Rolling back a multi-cluster application reverts the application for **all** target clusters and projects, not just the targets(s) affected by the upgrade issue. - -## Deleting a Multi-Cluster Application - -:::note Prerequisite: - -The `legacy` feature flag needs to be enabled. - -::: - -1. In the upper left corner, click **☰ > Multi-cluster Apps**. - -2. Choose the multi-cluster application you want to delete and click the **⋮ > Delete**. When deleting the multi-cluster application, all applications and namespaces are deleted in all of the target projects. - - :::note - - The applications in the target projects, that are created for a multi-cluster application, cannot be deleted individually. The applications can only be deleted when the multi-cluster application is deleted. - - ::: \ No newline at end of file From c59164ac5cf5f7b720863bbed66da00d9809c816 Mon Sep 17 00:00:00 2001 From: martyav Date: Fri, 5 Apr 2024 11:04:41 -0400 Subject: [PATCH 04/67] Include documentation on how to make UI extensions work in airgap mode in the official Rancher Docs --- .../rancher-extensions.md | 71 ++++++++++++++++++- 1 file changed, 69 insertions(+), 2 deletions(-) diff --git a/docs/integrations-in-rancher/rancher-extensions.md b/docs/integrations-in-rancher/rancher-extensions.md index 2c56082fc53..499d0f81711 100644 --- a/docs/integrations-in-rancher/rancher-extensions.md +++ b/docs/integrations-in-rancher/rancher-extensions.md @@ -53,9 +53,76 @@ In v2.7.0, the built-in extensions aren't displayed under the **Available** tab. ![Reload button](/img/reload-button.png) -### Importing and Installing Extensions in an Air-Gapped Environment +### Accessing Rancher UI Extensions in Air-Gapped Environments -1. Find the address of the container image repository that you want to import as an extension. Rancher provides some extensions, such as Kubewarden and Elemental, through the `ui-plugin-catalog` container image at https://hub.docker.com/r/rancher/ui-plugin-catalog/tags. You should import and use the latest tagged version of the image to ensure you receive the latest features and security updates. +Rancher provides some extensions, such as Kubewarden and Elemental, through the `ui-plugin-catalog` container image at https://hub.docker.com/r/rancher/ui-plugin-catalog/tags. If you're trying to install these extensions in an air-gapped environment, you must make the `ui-plugin-catalog` accessible. + +1. Mirror the `ui-plugin-catalog` image to a private registry: +```bash +export REGISTRY_ENDPOINT="my-private-registry.com" +docker pull rancher/ui-plugin-catalog:1.0.0 +docker tag rancher/ui-plugin-catalog:1.0.0 $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog:1.0.0 +docker push $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog:1.0.0 +``` +1. Use the mirrored image to create a Kubernetes [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/): +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: ui-plugin-catalog + namespace: cattle-ui-plugin-system + labels: + catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog +spec: + replicas: 1 + selector: + matchLabels: + catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog + template: + metadata: + namespace: cattle-ui-plugin-system + labels: + catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog + spec: + containers: + - name: server + image: my-private-registry.com/rancher/ui-plugin-catalog:1.0.0 + imagePullPolicy: Always + imagePullSecrets: + - name: my-registry-credentials +``` +1. Expose the deployment by creating a [ClusterIP service](https://kubernetes.io/docs/concepts/services-networking/service/#type-clusterip): +```yaml +apiVersion: v1 +kind: Service +metadata: + name: ui-plugin-catalog-svc + namespace: cattle-ui-plugin-system +spec: + ports: + - name: catalog-svc-port + port: 8080 + protocol: TCP + targetPort: 8080 + selector: + catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog + type: ClusterIP +``` +1. Create a [ClusterRepo](../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md) that targets the ClusterIP service: +```yaml +apiVersion: catalog.cattle.io/v1 +kind: ClusterRepo +metadata: + name: ui-plugin-catalog-repo +spec: + url: http://ui-plugin-catalog-svc.cattle-ui-plugin-system:8080 +``` + +After you successfully set up these resources, you can install the extensions from the `ui-plugin-charts` manifest into your air-gapped environment. + +### Importing and Installing Extensions in an Air-gapped Environment + +1. Find the address of the container image repository that you want to import as an extension. You should import and use the latest tagged version of the image to ensure you receive the latest features and security updates. * **(Optional)** If the container image is private: [Create](../how-to-guides/new-user-guides/kubernetes-resources-setup/secrets.md) a registry secret within the `cattle-ui-plugin-system` namespace. Enter the domain of the image address in the **Registry Domain Name** field. From 3bd7e533e80cdc0b12c8505ed0a2fea7da4c7e6e Mon Sep 17 00:00:00 2001 From: martyav Date: Fri, 5 Apr 2024 11:31:19 -0400 Subject: [PATCH 05/67] reorganize air-gapped subsections into single parent section --- .../rancher-extensions.md | 104 +++++++++--------- 1 file changed, 54 insertions(+), 50 deletions(-) diff --git a/docs/integrations-in-rancher/rancher-extensions.md b/docs/integrations-in-rancher/rancher-extensions.md index 499d0f81711..e3c29b18a3b 100644 --- a/docs/integrations-in-rancher/rancher-extensions.md +++ b/docs/integrations-in-rancher/rancher-extensions.md @@ -53,7 +53,60 @@ In v2.7.0, the built-in extensions aren't displayed under the **Available** tab. ![Reload button](/img/reload-button.png) -### Accessing Rancher UI Extensions in Air-Gapped Environments +## Updating and Upgrading Extensions + +1. Click **☰ > Extensions** under **Configuration**. +1. Select the **Updates** tab. +1. Click **Update**. + +If there is a new version of the extension, there will also be an **Update** button visible on the associated card for the extension in the **Available** tab. + +## Deleting Helm Charts + +1. Click **☰**, then click on the name of your local cluster. +1. From the sidebar, select **Apps > Installed Apps**. +1. Find the name of the chart you want to delete and select the checkbox next to it. +1. Click **Delete**. + +## Deleting Extension Repositories + +1. Click **☰ > Extensions** under **Configuration**. +1. On the top right, click **⋮ > Manage Repositories**. +1. Find the name of the extension repository you want to delete. Select the checkbox next to the repository name, then click **Delete**. + +## Deleting Extension Repository Container Images + +1. Click **☰**, then select **Extensions**, under **Configuration**. +1. On the top right, click **⋮ > Manage Extension Catalogs**. +1. Find the name of the container image you want to delete. Click **⋮ > Uninstall**. + +## Uninstalling Extensions + +There are two ways to uninstall or disable an extension: + +1. Under the **Installed** tab, click the **Uninstall** button on the extension you wish to remove. + + ![Uninstall extensions](/img/uninstall-extension.png) + +1. On the extensions management page, click **⋮ > Disable Extension Support**. This will disable all installed extensions. + + ![Disable extensions](/img/disable-extension-support.png) + +:::caution + +You must reload the page after disabling extensions or display issues may occur. + +::: + +## Developing Extensions + +To learn how to develop your own extensions, refer to the official [Getting Started](https://rancher.github.io/dashboard/extensions/extensions-getting-started) guide. + +## Working with Extensions in an Air-gapped Environment + +If you intend to work with extensions in an air-gapped environment, you must perform some extra steps before you can complete certain tasks. + +### Accessing Rancher UI Extensions in an Air-Gapped Environment Rancher provides some extensions, such as Kubewarden and Elemental, through the `ui-plugin-catalog` container image at https://hub.docker.com/r/rancher/ui-plugin-catalog/tags. If you're trying to install these extensions in an air-gapped environment, you must make the `ui-plugin-catalog` accessible. @@ -144,32 +197,6 @@ After you successfully set up these resources, you can install the extensions fr 1. Find the extension you just added, and click the **Install** button. -## Uninstalling Extensions - -There are two ways to uninstall or disable an extension: - -1. Under the **Installed** tab, click the **Uninstall** button on the extension you wish to remove. - - ![Uninstall extensions](/img/uninstall-extension.png) - -1. On the extensions management page, click **⋮ > Disable Extension Support**. This will disable all installed extensions. - - ![Disable extensions](/img/disable-extension-support.png) - -:::caution - -You must reload the page after disabling extensions or display issues may occur. - -::: - -## Updating and Upgrading Extensions - -1. Click **☰ > Extensions** under **Configuration**. -1. Select the **Updates** tab. -1. Click **Update**. - -If there is a new version of the extension, there will also be an **Update** button visible on the associated card for the extension in the **Available** tab. - ### Updating and Upgrading an Extensions Repository in an Air-gapped Environment Extensions repositories that aren't air-gapped are automatically updated. If the repository is air-gapped, you must update it manually. @@ -186,26 +213,3 @@ After you mirror the latest changes, follow these steps: 1. Click **⋮ > Edit config**. 1. Update the **Container Image** field within the deployment's container with the latest image. 1. Click **Save**. - -## Deleting Helm Charts - -1. Click **☰**, then click on the name of your local cluster. -1. From the sidebar, select **Apps > Installed Apps**. -1. Find the name of the chart you want to delete and select the checkbox next to it. -1. Click **Delete**. - -## Deleting Extension Repositories - -1. Click **☰ > Extensions** under **Configuration**. -1. On the top right, click **⋮ > Manage Repositories**. -1. Find the name of the extension repository you want to delete. Select the checkbox next to the repository name, then click **Delete**. - -## Deleting Extension Repository Container Images - -1. Click **☰**, then select **Extensions**, under **Configuration**. -1. On the top right, click **⋮ > Manage Extension Catalogs**. -1. Find the name of the container image you want to delete. Click **⋮ > Uninstall**. - -## Developing Extensions - -To learn how to develop your own extensions, refer to the official [Getting Started](https://rancher.github.io/dashboard/extensions/extensions-getting-started) guide. From 35702127cb7c91ac85007aa35050f226bc820fde Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Fri, 5 Apr 2024 13:38:17 -0400 Subject: [PATCH 06/67] Apply suggestions from code review Co-authored-by: Lucas Saintarbor --- .../rancher-extensions.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/integrations-in-rancher/rancher-extensions.md b/docs/integrations-in-rancher/rancher-extensions.md index e3c29b18a3b..55e48ea9aeb 100644 --- a/docs/integrations-in-rancher/rancher-extensions.md +++ b/docs/integrations-in-rancher/rancher-extensions.md @@ -78,7 +78,7 @@ If there is a new version of the extension, there will also be an **Update** but 1. Click **☰**, then select **Extensions**, under **Configuration**. 1. On the top right, click **⋮ > Manage Extension Catalogs**. -1. Find the name of the container image you want to delete. Click **⋮ > Uninstall**. +1. Find the name of the container image you want to delete, then click **⋮ > Uninstall**. ## Uninstalling Extensions @@ -111,12 +111,12 @@ If you intend to work with extensions in an air-gapped environment, you must per Rancher provides some extensions, such as Kubewarden and Elemental, through the `ui-plugin-catalog` container image at https://hub.docker.com/r/rancher/ui-plugin-catalog/tags. If you're trying to install these extensions in an air-gapped environment, you must make the `ui-plugin-catalog` accessible. 1. Mirror the `ui-plugin-catalog` image to a private registry: -```bash -export REGISTRY_ENDPOINT="my-private-registry.com" -docker pull rancher/ui-plugin-catalog:1.0.0 -docker tag rancher/ui-plugin-catalog:1.0.0 $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog:1.0.0 -docker push $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog:1.0.0 -``` + + ```bash + export REGISTRY_ENDPOINT="my-private-registry.com" + docker pull rancher/ui-plugin-catalog:1.0.0 + docker tag rancher/ui-plugin-catalog:1.0.0 $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog:1.0.0 + docker push $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog:1.0.0 1. Use the mirrored image to create a Kubernetes [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/): ```yaml apiVersion: apps/v1 @@ -187,7 +187,7 @@ After you successfully set up these resources, you can install the extensions fr 1. Enter the image address in the **Catalog Image Reference** field. - * **(Optional)** If the container image is private: Select the secret you just created from the **Pull Secrets** drop-down menu. + * **(Optional)** If the container image is private, select the secret you just created from the **Pull Secrets** drop-down menu. 1. Click **Load**. The extension will now be **Pending**. From ebdaa578d2b7bedd248472e3f40ae21a079dcc5d Mon Sep 17 00:00:00 2001 From: martyav Date: Fri, 5 Apr 2024 13:50:54 -0400 Subject: [PATCH 07/67] spacing --- .../rancher-extensions.md | 112 ++++++++---------- 1 file changed, 51 insertions(+), 61 deletions(-) diff --git a/docs/integrations-in-rancher/rancher-extensions.md b/docs/integrations-in-rancher/rancher-extensions.md index 55e48ea9aeb..ea732372654 100644 --- a/docs/integrations-in-rancher/rancher-extensions.md +++ b/docs/integrations-in-rancher/rancher-extensions.md @@ -117,84 +117,74 @@ Rancher provides some extensions, such as Kubewarden and Elemental, through the docker pull rancher/ui-plugin-catalog:1.0.0 docker tag rancher/ui-plugin-catalog:1.0.0 $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog:1.0.0 docker push $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog:1.0.0 -1. Use the mirrored image to create a Kubernetes [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/): -```yaml -apiVersion: apps/v1 -kind: Deployment -metadata: - name: ui-plugin-catalog - namespace: cattle-ui-plugin-system - labels: - catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog -spec: - replicas: 1 - selector: - matchLabels: +2. Use the mirrored image to create a Kubernetes [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/): + ```yaml + apiVersion: apps/v1 + kind: Deployment + metadata: + name: ui-plugin-catalog + namespace: cattle-ui-plugin-system + labels: catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog - template: - metadata: - namespace: cattle-ui-plugin-system - labels: + spec: + replicas: 1 + selector: + matchLabels: catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog - spec: - containers: - - name: server - image: my-private-registry.com/rancher/ui-plugin-catalog:1.0.0 - imagePullPolicy: Always - imagePullSecrets: - - name: my-registry-credentials -``` -1. Expose the deployment by creating a [ClusterIP service](https://kubernetes.io/docs/concepts/services-networking/service/#type-clusterip): -```yaml -apiVersion: v1 -kind: Service -metadata: - name: ui-plugin-catalog-svc - namespace: cattle-ui-plugin-system -spec: - ports: - - name: catalog-svc-port - port: 8080 - protocol: TCP - targetPort: 8080 - selector: - catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog - type: ClusterIP -``` -1. Create a [ClusterRepo](../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md) that targets the ClusterIP service: -```yaml -apiVersion: catalog.cattle.io/v1 -kind: ClusterRepo -metadata: - name: ui-plugin-catalog-repo -spec: - url: http://ui-plugin-catalog-svc.cattle-ui-plugin-system:8080 -``` + template: + metadata: + namespace: cattle-ui-plugin-system + labels: + catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog + spec: + containers: + - name: server + image: my-private-registry.com/rancher/ui-plugin-catalog:1.0.0 + imagePullPolicy: Always + imagePullSecrets: + - name: my-registry-credentials + ``` +3. Expose the deployment by creating a [ClusterIP service](https://kubernetes.io/docs/concepts/services-networking/service/#type-clusterip): + ```yaml + apiVersion: v1 + kind: Service + metadata: + name: ui-plugin-catalog-svc + namespace: cattle-ui-plugin-system + spec: + ports: + - name: catalog-svc-port + port: 8080 + protocol: TCP + targetPort: 8080 + selector: + catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog + type: ClusterIP + ``` +4. Create a [ClusterRepo](../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md) that targets the ClusterIP service: + ```yaml + apiVersion: catalog.cattle.io/v1 + kind: ClusterRepo + metadata: + name: ui-plugin-catalog-repo + spec: + url: http://ui-plugin-catalog-svc.cattle-ui-plugin-system:8080 + ``` After you successfully set up these resources, you can install the extensions from the `ui-plugin-charts` manifest into your air-gapped environment. ### Importing and Installing Extensions in an Air-gapped Environment 1. Find the address of the container image repository that you want to import as an extension. You should import and use the latest tagged version of the image to ensure you receive the latest features and security updates. - * **(Optional)** If the container image is private: [Create](../how-to-guides/new-user-guides/kubernetes-resources-setup/secrets.md) a registry secret within the `cattle-ui-plugin-system` namespace. Enter the domain of the image address in the **Registry Domain Name** field. - 1. Click **☰**, then select **Extensions**, under **Configuration**. - 1. On the top right, click **⋮ > Manage Extension Catalogs**. - 1. Select the **Import Extension Catalog** button. - 1. Enter the image address in the **Catalog Image Reference** field. - * **(Optional)** If the container image is private, select the secret you just created from the **Pull Secrets** drop-down menu. - 1. Click **Load**. The extension will now be **Pending**. - 1. Return to the **Extensions** page. - 1. Select the **Available** tab, and click the **Reload** button to make sure that the list of extensions is up to date. - 1. Find the extension you just added, and click the **Install** button. ### Updating and Upgrading an Extensions Repository in an Air-gapped Environment From 297b515f4d0901b672feeb68c4a95f1136c82280 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Fri, 5 Apr 2024 10:54:08 -0700 Subject: [PATCH 08/67] Updating after review for syntax/wording and moving pre-reqs into top intro section. Signed-off-by: Sunil Singh --- .../cluster-api/capi.md | 8 +- .../cluster-api/overview.md | 86 ++++++++----------- .../cluster-api/capi.md | 8 +- .../cluster-api/overview.md | 86 ++++++++----------- .../cluster-api/capi.md | 8 +- .../cluster-api/overview.md | 86 ++++++++----------- 6 files changed, 126 insertions(+), 156 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/capi.md b/docs/integrations-in-rancher/cluster-api/capi.md index 53ec6f4a66f..68d2e67665b 100644 --- a/docs/integrations-in-rancher/cluster-api/capi.md +++ b/docs/integrations-in-rancher/cluster-api/capi.md @@ -6,9 +6,9 @@ title: Cluster API with Rancher Turtles -Rancher Turtles is a [Rancher extension](../rancher-extensions.md) that provides integration between your Cluster API (CAPI) and Rancher to manage the lifecycle of provisioned Kubernetes clusters. The main functionality is outlined below: +Rancher Turtles is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: -- Importing CAPI clusters into Rancher: Installing the Rancher Cluster Agent in CAPI provisioned clusters. -- Configuration support for the CAPI Operator. +- Import CAPI clusters into Rancher, by installing the Rancher Cluster Agent in CAPI provisioned clusters. +- Configure the CAPI Operator. -The [Overview](./overview.md) section outlines installation options, the main architecture, and a brief demo. For more detailed information, please view the [Rancher Turtles documentation](https://docs.rancher-turtles.com/docs/intro). +The [Overview](./overview.md) section outlines installation options, Rancher Turtles architecture, and a brief demo. For more details, see the [Rancher Turtles documentation](https://turtles.docs.rancher.com/). diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 4320dc87774..2e050ee1804 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -12,39 +12,39 @@ title: Overview ## Installing the Rancher Turtles Operator -### Via Rancher Dashboard - -This is the recommended option for installing Rancher Turtles. - -Using the Rancher UI, and by adding the Turtles repository, we can let Rancher process the installation and configuration of the Cluster API Extension. +You can install the Rancher Turtles operator via the Rancher UI, or with Helm. The first method is recommended for most environments. :::caution -If you already have the Cluster API Operator installed in your cluster, you need to use the [manual helm installation method](./overview.md#via-helm-install) instead. +If you already have the Cluster API Operator installed in your cluster, you must use the [manual helm installation method](./overview.md#via-helm-install). ::: :::info prerequisite -Before installing Rancher Turtles in your Rancher environment, Rancher's `embedded-cluster-api` functionality must be disabled. This includes also cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. +Before installing Rancher Turtles in your Rancher environment, you must disable Rancher's `embedded-cluster-api` functionality. This also includes cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. -To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that applies these changes which are outlined below if manually removing: -- Disable the `embedded-cluster-api` feature in Rancher. -- Delete the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. +To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that removes the following: +- Disables the `embedded-cluster-api` feature in Rancher. +- Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. ::: +### Via Rancher UI + +By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the Cluster API Extension. + #### Installation -- From your browser, access Rancher Manager and explore the **local** cluster. -- Using the left navigation panel, go to `Apps` -> `Repositories`. -- Click `Create` to add a new repository. +- Click **☰**. Under **Explore Cluster** in the left navigation menu, select **local**. +- In the left navigation menu of the **Cluster Dashboard**, select **Apps > Repositories**. +- Click **Create** to add a new repository. - Enter the following: - - **Name**: `turtles`. - - **Index URL**: https://rancher.github.io/turtles. -- Wait for the `turtles` repository to have a status of `Active`. -- Go to `Apps` -> `Charts`. -- Filter for `turtles`. -- Click `Rancher Turtles - the Cluster API Extension` -- Click `Install` -> `Next` -> `Install`. + - **Name**: turtles + - **Index URL**: https://rancher.github.io/turtles +- Wait until the new repository has a status of `Active`. +- In the left navigation menu, select **Apps > Charts**. +- Enter "turtles" into the search filter to find the Turtles chart. +- Click **Rancher Turtles - the Cluster API Extension**. +- Click **Install > Next > Install**. -This will use the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or, alternatively, you can opt for the [manual installation via Helm](./overview.md#via-helm-install). If you are interested in learning more about the available values, you can check the Rancher Turtles [Helm chart reference guide](https://docs.rancher-turtles.com/docs/reference-guides/rancher-turtles-chart/values). +This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#via-helm-install). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). The installation may take a few minutes and once completed you will see the following new deployments in the cluster: - `rancher-turtles-system/rancher-turtles-controller-manager` @@ -53,32 +53,24 @@ The installation may take a few minutes and once completed you will see the foll ### Via Helm Install -This section walks through different Helm installation options for Rancher Turtles depending on CAPI operator use: +The Helm installation options for Rancher Turtles has two paths depending on CAPI operator use: - [Install Rancher Turtles with Cluster API Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency). - [Install Rancher Turtles without Cluster API Operator](#install-rancher-turtles-without-cluster-api-operator-as-a-helm-dependency). -The Cluster API Operator is required for installing Rancher Turtles and you can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. We recommend [installing as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) for the sake of simplicity, but the best option may depend on your specific configuration. +The Cluster API Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. CAPI Operator allows for handling the lifecycle of Cluster API providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). -:::info prerequisite -Before installing Rancher Turtles in your Rancher environment, Rancher's `embedded-cluster-api` functionality must be disabled. This includes also cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. - -To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that applies these changes which are outlined below: -- Disable the `embedded-cluster-api` feature in Rancher. -- Delete the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. -::: - #### Install Rancher Turtles with `Cluster API Operator` as a Helm dependency -The `rancher-turtles` chart is available in https://rancher.github.io/turtles and this Helm repository must be added before proceeding with the installation: +The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: ```bash helm repo add turtles https://rancher.github.io/turtles helm repo update ``` -As mentioned before, installing Rancher Turtles requires the [Cluster API Operator](https://github.com/kubernetes-sigs/cluster-api-operator) and the Helm chart can handle its installation automatically with a minimum set of flags: +As mentioned before, installing Rancher Turtles requires the [Cluster API Operator](https://github.com/kubernetes-sigs/cluster-api-operator). The Helm chart can automatically install it with a minimal set of flags: ```bash helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ @@ -93,19 +85,19 @@ This operation could take a few minutes and once completed you can review the in - `capi-operator` :::note -- If `cert-manager` is already available in the cluster, you can disable its installation as a Rancher Turtles dependency to avoid conflicts: +- If `cert-manager` is already available in the cluster, to avoid conflicts, disable its installation as a Rancher Turtles dependency: `--set cluster-api-operator.cert-manager.enabled=false` -- For a list of Rancher Turtles versions, refer to [Releases page](https://github.com/rancher/turtles/releases). +- For a list of Rancher Turtles versions, refer to the [Turtles release page](https://github.com/rancher/turtles/releases). ::: This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional Cluster API functionality. -If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled. +If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including enabling the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL`. ```bash helm install ... # Passing secret name and namespace for additional environment variables - --set cluster-api-operator.cluster-api.configSecret.name= + --set cluster-api-operator.cluster-api.configSecret.name= ``` The following is an example of a user-managed secret `cluster-api-operator.cluster-api.configSecret.name=variables` with `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` feature flags set and an extra custom variable: @@ -125,23 +117,23 @@ stringData: ``` :::info -For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://docs.rancher-turtles.com/docs/reference-guides/rancher-turtles-chart/values) +For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) ::: #### Install Rancher Turtles without `Cluster API Operator` as a Helm dependency :::note -Remember that if you opt for this installation option, you will need to manage the Cluster API Operator installation yourself. You can follow the [CAPI Operator guide](https://docs.rancher-turtles.com/docs/next/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. +Remember that if you opt for this installation option, you must manage the Cluster API Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. ::: -The `rancher-turtles` chart is available in https://rancher.github.io/turtles and this Helm repository must be added before proceeding with the installation: +Add the `rancher-turtles` chart from the in https://rancher.github.io/turtles Helm repository before installation: ```bash helm repo add turtles https://rancher.github.io/turtles helm repo update ``` -and then it can be installed into the `rancher-turtles-system` namespace with: +Install the chart into the `rancher-turtles-system` namespace: ```bash helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 @@ -152,24 +144,22 @@ helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 --dependency-update ``` -As you can see, we are telling Helm to ignore installing `cluster-api-operator` as a dependency. +The previous commands tell Helm to ignore installing `cluster-api-operator` as a dependency. This operation could take a few minutes and once completed you can review the installed controller listed below: - `rancher-turtles-controller` ## Uninstalling Rancher Turtles -This section gives an overview of the Rancher Turtles uninstallation process. - :::caution When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the Cluster API Operator cleanup. This includes cleaning up Cluster API Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. -To simplify uninstalling Rancher Turtles (via Rancher Manager or helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that applies these changes that are outlined below: +To simplify uninstalling Rancher Turtles (via Rancher Manager or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. - Deletes the CAPI `deployments` that are no longer needed. ::: -To uninstall the Rancher Turtles Extension use the following helm command: +To uninstall Rancher Turtles: ```bash helm uninstall -n rancher-turtles-system rancher-turtles --cascade foreground --wait @@ -181,7 +171,7 @@ This may take a few minutes to complete. Remember that, if you use a different name for the installation or a different namespace, you may need to customize the command for your specific configuration. ::: -Once uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: +Once Rancher Turtles is uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: 1. Create a `feature.yaml` file, with `embedded-cluster-api` set to true: ```yaml title="feature.yaml" @@ -207,4 +197,4 @@ Below is a visual representation of the key components of Rancher Turtles and th [SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. -Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, please visit the [Rancher Turtles Security](https://docs.rancher-turtles.com/docs/security/slsa) document. +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/docs/security/slsa) document. diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/capi.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/capi.md index 53ec6f4a66f..68d2e67665b 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/capi.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/capi.md @@ -6,9 +6,9 @@ title: Cluster API with Rancher Turtles -Rancher Turtles is a [Rancher extension](../rancher-extensions.md) that provides integration between your Cluster API (CAPI) and Rancher to manage the lifecycle of provisioned Kubernetes clusters. The main functionality is outlined below: +Rancher Turtles is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: -- Importing CAPI clusters into Rancher: Installing the Rancher Cluster Agent in CAPI provisioned clusters. -- Configuration support for the CAPI Operator. +- Import CAPI clusters into Rancher, by installing the Rancher Cluster Agent in CAPI provisioned clusters. +- Configure the CAPI Operator. -The [Overview](./overview.md) section outlines installation options, the main architecture, and a brief demo. For more detailed information, please view the [Rancher Turtles documentation](https://docs.rancher-turtles.com/docs/intro). +The [Overview](./overview.md) section outlines installation options, Rancher Turtles architecture, and a brief demo. For more details, see the [Rancher Turtles documentation](https://turtles.docs.rancher.com/). diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index 4320dc87774..2e050ee1804 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -12,39 +12,39 @@ title: Overview ## Installing the Rancher Turtles Operator -### Via Rancher Dashboard - -This is the recommended option for installing Rancher Turtles. - -Using the Rancher UI, and by adding the Turtles repository, we can let Rancher process the installation and configuration of the Cluster API Extension. +You can install the Rancher Turtles operator via the Rancher UI, or with Helm. The first method is recommended for most environments. :::caution -If you already have the Cluster API Operator installed in your cluster, you need to use the [manual helm installation method](./overview.md#via-helm-install) instead. +If you already have the Cluster API Operator installed in your cluster, you must use the [manual helm installation method](./overview.md#via-helm-install). ::: :::info prerequisite -Before installing Rancher Turtles in your Rancher environment, Rancher's `embedded-cluster-api` functionality must be disabled. This includes also cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. +Before installing Rancher Turtles in your Rancher environment, you must disable Rancher's `embedded-cluster-api` functionality. This also includes cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. -To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that applies these changes which are outlined below if manually removing: -- Disable the `embedded-cluster-api` feature in Rancher. -- Delete the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. +To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that removes the following: +- Disables the `embedded-cluster-api` feature in Rancher. +- Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. ::: +### Via Rancher UI + +By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the Cluster API Extension. + #### Installation -- From your browser, access Rancher Manager and explore the **local** cluster. -- Using the left navigation panel, go to `Apps` -> `Repositories`. -- Click `Create` to add a new repository. +- Click **☰**. Under **Explore Cluster** in the left navigation menu, select **local**. +- In the left navigation menu of the **Cluster Dashboard**, select **Apps > Repositories**. +- Click **Create** to add a new repository. - Enter the following: - - **Name**: `turtles`. - - **Index URL**: https://rancher.github.io/turtles. -- Wait for the `turtles` repository to have a status of `Active`. -- Go to `Apps` -> `Charts`. -- Filter for `turtles`. -- Click `Rancher Turtles - the Cluster API Extension` -- Click `Install` -> `Next` -> `Install`. + - **Name**: turtles + - **Index URL**: https://rancher.github.io/turtles +- Wait until the new repository has a status of `Active`. +- In the left navigation menu, select **Apps > Charts**. +- Enter "turtles" into the search filter to find the Turtles chart. +- Click **Rancher Turtles - the Cluster API Extension**. +- Click **Install > Next > Install**. -This will use the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or, alternatively, you can opt for the [manual installation via Helm](./overview.md#via-helm-install). If you are interested in learning more about the available values, you can check the Rancher Turtles [Helm chart reference guide](https://docs.rancher-turtles.com/docs/reference-guides/rancher-turtles-chart/values). +This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#via-helm-install). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). The installation may take a few minutes and once completed you will see the following new deployments in the cluster: - `rancher-turtles-system/rancher-turtles-controller-manager` @@ -53,32 +53,24 @@ The installation may take a few minutes and once completed you will see the foll ### Via Helm Install -This section walks through different Helm installation options for Rancher Turtles depending on CAPI operator use: +The Helm installation options for Rancher Turtles has two paths depending on CAPI operator use: - [Install Rancher Turtles with Cluster API Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency). - [Install Rancher Turtles without Cluster API Operator](#install-rancher-turtles-without-cluster-api-operator-as-a-helm-dependency). -The Cluster API Operator is required for installing Rancher Turtles and you can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. We recommend [installing as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) for the sake of simplicity, but the best option may depend on your specific configuration. +The Cluster API Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. CAPI Operator allows for handling the lifecycle of Cluster API providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). -:::info prerequisite -Before installing Rancher Turtles in your Rancher environment, Rancher's `embedded-cluster-api` functionality must be disabled. This includes also cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. - -To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that applies these changes which are outlined below: -- Disable the `embedded-cluster-api` feature in Rancher. -- Delete the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. -::: - #### Install Rancher Turtles with `Cluster API Operator` as a Helm dependency -The `rancher-turtles` chart is available in https://rancher.github.io/turtles and this Helm repository must be added before proceeding with the installation: +The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: ```bash helm repo add turtles https://rancher.github.io/turtles helm repo update ``` -As mentioned before, installing Rancher Turtles requires the [Cluster API Operator](https://github.com/kubernetes-sigs/cluster-api-operator) and the Helm chart can handle its installation automatically with a minimum set of flags: +As mentioned before, installing Rancher Turtles requires the [Cluster API Operator](https://github.com/kubernetes-sigs/cluster-api-operator). The Helm chart can automatically install it with a minimal set of flags: ```bash helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ @@ -93,19 +85,19 @@ This operation could take a few minutes and once completed you can review the in - `capi-operator` :::note -- If `cert-manager` is already available in the cluster, you can disable its installation as a Rancher Turtles dependency to avoid conflicts: +- If `cert-manager` is already available in the cluster, to avoid conflicts, disable its installation as a Rancher Turtles dependency: `--set cluster-api-operator.cert-manager.enabled=false` -- For a list of Rancher Turtles versions, refer to [Releases page](https://github.com/rancher/turtles/releases). +- For a list of Rancher Turtles versions, refer to the [Turtles release page](https://github.com/rancher/turtles/releases). ::: This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional Cluster API functionality. -If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled. +If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including enabling the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL`. ```bash helm install ... # Passing secret name and namespace for additional environment variables - --set cluster-api-operator.cluster-api.configSecret.name= + --set cluster-api-operator.cluster-api.configSecret.name= ``` The following is an example of a user-managed secret `cluster-api-operator.cluster-api.configSecret.name=variables` with `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` feature flags set and an extra custom variable: @@ -125,23 +117,23 @@ stringData: ``` :::info -For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://docs.rancher-turtles.com/docs/reference-guides/rancher-turtles-chart/values) +For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) ::: #### Install Rancher Turtles without `Cluster API Operator` as a Helm dependency :::note -Remember that if you opt for this installation option, you will need to manage the Cluster API Operator installation yourself. You can follow the [CAPI Operator guide](https://docs.rancher-turtles.com/docs/next/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. +Remember that if you opt for this installation option, you must manage the Cluster API Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. ::: -The `rancher-turtles` chart is available in https://rancher.github.io/turtles and this Helm repository must be added before proceeding with the installation: +Add the `rancher-turtles` chart from the in https://rancher.github.io/turtles Helm repository before installation: ```bash helm repo add turtles https://rancher.github.io/turtles helm repo update ``` -and then it can be installed into the `rancher-turtles-system` namespace with: +Install the chart into the `rancher-turtles-system` namespace: ```bash helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 @@ -152,24 +144,22 @@ helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 --dependency-update ``` -As you can see, we are telling Helm to ignore installing `cluster-api-operator` as a dependency. +The previous commands tell Helm to ignore installing `cluster-api-operator` as a dependency. This operation could take a few minutes and once completed you can review the installed controller listed below: - `rancher-turtles-controller` ## Uninstalling Rancher Turtles -This section gives an overview of the Rancher Turtles uninstallation process. - :::caution When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the Cluster API Operator cleanup. This includes cleaning up Cluster API Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. -To simplify uninstalling Rancher Turtles (via Rancher Manager or helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that applies these changes that are outlined below: +To simplify uninstalling Rancher Turtles (via Rancher Manager or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. - Deletes the CAPI `deployments` that are no longer needed. ::: -To uninstall the Rancher Turtles Extension use the following helm command: +To uninstall Rancher Turtles: ```bash helm uninstall -n rancher-turtles-system rancher-turtles --cascade foreground --wait @@ -181,7 +171,7 @@ This may take a few minutes to complete. Remember that, if you use a different name for the installation or a different namespace, you may need to customize the command for your specific configuration. ::: -Once uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: +Once Rancher Turtles is uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: 1. Create a `feature.yaml` file, with `embedded-cluster-api` set to true: ```yaml title="feature.yaml" @@ -207,4 +197,4 @@ Below is a visual representation of the key components of Rancher Turtles and th [SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. -Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, please visit the [Rancher Turtles Security](https://docs.rancher-turtles.com/docs/security/slsa) document. +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/docs/security/slsa) document. diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md index 53ec6f4a66f..68d2e67665b 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md @@ -6,9 +6,9 @@ title: Cluster API with Rancher Turtles -Rancher Turtles is a [Rancher extension](../rancher-extensions.md) that provides integration between your Cluster API (CAPI) and Rancher to manage the lifecycle of provisioned Kubernetes clusters. The main functionality is outlined below: +Rancher Turtles is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: -- Importing CAPI clusters into Rancher: Installing the Rancher Cluster Agent in CAPI provisioned clusters. -- Configuration support for the CAPI Operator. +- Import CAPI clusters into Rancher, by installing the Rancher Cluster Agent in CAPI provisioned clusters. +- Configure the CAPI Operator. -The [Overview](./overview.md) section outlines installation options, the main architecture, and a brief demo. For more detailed information, please view the [Rancher Turtles documentation](https://docs.rancher-turtles.com/docs/intro). +The [Overview](./overview.md) section outlines installation options, Rancher Turtles architecture, and a brief demo. For more details, see the [Rancher Turtles documentation](https://turtles.docs.rancher.com/). diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index 4320dc87774..2e050ee1804 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -12,39 +12,39 @@ title: Overview ## Installing the Rancher Turtles Operator -### Via Rancher Dashboard - -This is the recommended option for installing Rancher Turtles. - -Using the Rancher UI, and by adding the Turtles repository, we can let Rancher process the installation and configuration of the Cluster API Extension. +You can install the Rancher Turtles operator via the Rancher UI, or with Helm. The first method is recommended for most environments. :::caution -If you already have the Cluster API Operator installed in your cluster, you need to use the [manual helm installation method](./overview.md#via-helm-install) instead. +If you already have the Cluster API Operator installed in your cluster, you must use the [manual helm installation method](./overview.md#via-helm-install). ::: :::info prerequisite -Before installing Rancher Turtles in your Rancher environment, Rancher's `embedded-cluster-api` functionality must be disabled. This includes also cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. +Before installing Rancher Turtles in your Rancher environment, you must disable Rancher's `embedded-cluster-api` functionality. This also includes cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. -To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that applies these changes which are outlined below if manually removing: -- Disable the `embedded-cluster-api` feature in Rancher. -- Delete the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. +To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that removes the following: +- Disables the `embedded-cluster-api` feature in Rancher. +- Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. ::: +### Via Rancher UI + +By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the Cluster API Extension. + #### Installation -- From your browser, access Rancher Manager and explore the **local** cluster. -- Using the left navigation panel, go to `Apps` -> `Repositories`. -- Click `Create` to add a new repository. +- Click **☰**. Under **Explore Cluster** in the left navigation menu, select **local**. +- In the left navigation menu of the **Cluster Dashboard**, select **Apps > Repositories**. +- Click **Create** to add a new repository. - Enter the following: - - **Name**: `turtles`. - - **Index URL**: https://rancher.github.io/turtles. -- Wait for the `turtles` repository to have a status of `Active`. -- Go to `Apps` -> `Charts`. -- Filter for `turtles`. -- Click `Rancher Turtles - the Cluster API Extension` -- Click `Install` -> `Next` -> `Install`. + - **Name**: turtles + - **Index URL**: https://rancher.github.io/turtles +- Wait until the new repository has a status of `Active`. +- In the left navigation menu, select **Apps > Charts**. +- Enter "turtles" into the search filter to find the Turtles chart. +- Click **Rancher Turtles - the Cluster API Extension**. +- Click **Install > Next > Install**. -This will use the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or, alternatively, you can opt for the [manual installation via Helm](./overview.md#via-helm-install). If you are interested in learning more about the available values, you can check the Rancher Turtles [Helm chart reference guide](https://docs.rancher-turtles.com/docs/reference-guides/rancher-turtles-chart/values). +This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#via-helm-install). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). The installation may take a few minutes and once completed you will see the following new deployments in the cluster: - `rancher-turtles-system/rancher-turtles-controller-manager` @@ -53,32 +53,24 @@ The installation may take a few minutes and once completed you will see the foll ### Via Helm Install -This section walks through different Helm installation options for Rancher Turtles depending on CAPI operator use: +The Helm installation options for Rancher Turtles has two paths depending on CAPI operator use: - [Install Rancher Turtles with Cluster API Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency). - [Install Rancher Turtles without Cluster API Operator](#install-rancher-turtles-without-cluster-api-operator-as-a-helm-dependency). -The Cluster API Operator is required for installing Rancher Turtles and you can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. We recommend [installing as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) for the sake of simplicity, but the best option may depend on your specific configuration. +The Cluster API Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. CAPI Operator allows for handling the lifecycle of Cluster API providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). -:::info prerequisite -Before installing Rancher Turtles in your Rancher environment, Rancher's `embedded-cluster-api` functionality must be disabled. This includes also cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. - -To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that applies these changes which are outlined below: -- Disable the `embedded-cluster-api` feature in Rancher. -- Delete the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. -::: - #### Install Rancher Turtles with `Cluster API Operator` as a Helm dependency -The `rancher-turtles` chart is available in https://rancher.github.io/turtles and this Helm repository must be added before proceeding with the installation: +The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: ```bash helm repo add turtles https://rancher.github.io/turtles helm repo update ``` -As mentioned before, installing Rancher Turtles requires the [Cluster API Operator](https://github.com/kubernetes-sigs/cluster-api-operator) and the Helm chart can handle its installation automatically with a minimum set of flags: +As mentioned before, installing Rancher Turtles requires the [Cluster API Operator](https://github.com/kubernetes-sigs/cluster-api-operator). The Helm chart can automatically install it with a minimal set of flags: ```bash helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ @@ -93,19 +85,19 @@ This operation could take a few minutes and once completed you can review the in - `capi-operator` :::note -- If `cert-manager` is already available in the cluster, you can disable its installation as a Rancher Turtles dependency to avoid conflicts: +- If `cert-manager` is already available in the cluster, to avoid conflicts, disable its installation as a Rancher Turtles dependency: `--set cluster-api-operator.cert-manager.enabled=false` -- For a list of Rancher Turtles versions, refer to [Releases page](https://github.com/rancher/turtles/releases). +- For a list of Rancher Turtles versions, refer to the [Turtles release page](https://github.com/rancher/turtles/releases). ::: This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional Cluster API functionality. -If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled. +If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including enabling the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL`. ```bash helm install ... # Passing secret name and namespace for additional environment variables - --set cluster-api-operator.cluster-api.configSecret.name= + --set cluster-api-operator.cluster-api.configSecret.name= ``` The following is an example of a user-managed secret `cluster-api-operator.cluster-api.configSecret.name=variables` with `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` feature flags set and an extra custom variable: @@ -125,23 +117,23 @@ stringData: ``` :::info -For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://docs.rancher-turtles.com/docs/reference-guides/rancher-turtles-chart/values) +For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) ::: #### Install Rancher Turtles without `Cluster API Operator` as a Helm dependency :::note -Remember that if you opt for this installation option, you will need to manage the Cluster API Operator installation yourself. You can follow the [CAPI Operator guide](https://docs.rancher-turtles.com/docs/next/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. +Remember that if you opt for this installation option, you must manage the Cluster API Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. ::: -The `rancher-turtles` chart is available in https://rancher.github.io/turtles and this Helm repository must be added before proceeding with the installation: +Add the `rancher-turtles` chart from the in https://rancher.github.io/turtles Helm repository before installation: ```bash helm repo add turtles https://rancher.github.io/turtles helm repo update ``` -and then it can be installed into the `rancher-turtles-system` namespace with: +Install the chart into the `rancher-turtles-system` namespace: ```bash helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 @@ -152,24 +144,22 @@ helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 --dependency-update ``` -As you can see, we are telling Helm to ignore installing `cluster-api-operator` as a dependency. +The previous commands tell Helm to ignore installing `cluster-api-operator` as a dependency. This operation could take a few minutes and once completed you can review the installed controller listed below: - `rancher-turtles-controller` ## Uninstalling Rancher Turtles -This section gives an overview of the Rancher Turtles uninstallation process. - :::caution When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the Cluster API Operator cleanup. This includes cleaning up Cluster API Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. -To simplify uninstalling Rancher Turtles (via Rancher Manager or helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that applies these changes that are outlined below: +To simplify uninstalling Rancher Turtles (via Rancher Manager or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. - Deletes the CAPI `deployments` that are no longer needed. ::: -To uninstall the Rancher Turtles Extension use the following helm command: +To uninstall Rancher Turtles: ```bash helm uninstall -n rancher-turtles-system rancher-turtles --cascade foreground --wait @@ -181,7 +171,7 @@ This may take a few minutes to complete. Remember that, if you use a different name for the installation or a different namespace, you may need to customize the command for your specific configuration. ::: -Once uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: +Once Rancher Turtles is uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: 1. Create a `feature.yaml` file, with `embedded-cluster-api` set to true: ```yaml title="feature.yaml" @@ -207,4 +197,4 @@ Below is a visual representation of the key components of Rancher Turtles and th [SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. -Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, please visit the [Rancher Turtles Security](https://docs.rancher-turtles.com/docs/security/slsa) document. +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/docs/security/slsa) document. From 7cb319d340191d89b3449602b66bd3ec19cf2162 Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Fri, 5 Apr 2024 15:56:51 -0400 Subject: [PATCH 09/67] Apply suggestions from code review Co-authored-by: Billy Tat Co-authored-by: Lucas Saintarbor --- .../rancher-extensions.md | 22 +++++++++---------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/integrations-in-rancher/rancher-extensions.md b/docs/integrations-in-rancher/rancher-extensions.md index ea732372654..18fa4f2e549 100644 --- a/docs/integrations-in-rancher/rancher-extensions.md +++ b/docs/integrations-in-rancher/rancher-extensions.md @@ -61,7 +61,7 @@ In v2.7.0, the built-in extensions aren't displayed under the **Available** tab. If there is a new version of the extension, there will also be an **Update** button visible on the associated card for the extension in the **Available** tab. -## Deleting Helm Charts +## Deleting Extensions 1. Click **☰**, then click on the name of your local cluster. 1. From the sidebar, select **Apps > Installed Apps**. @@ -108,15 +108,15 @@ If you intend to work with extensions in an air-gapped environment, you must per ### Accessing Rancher UI Extensions in an Air-Gapped Environment -Rancher provides some extensions, such as Kubewarden and Elemental, through the `ui-plugin-catalog` container image at https://hub.docker.com/r/rancher/ui-plugin-catalog/tags. If you're trying to install these extensions in an air-gapped environment, you must make the `ui-plugin-catalog` accessible. +Rancher provides some extensions, such as Kubewarden and Elemental, through the `ui-plugin-catalog` container image at https://hub.docker.com/r/rancher/ui-plugin-catalog/tags. If you're trying to install these extensions in an air-gapped environment, you must make the `ui-plugin-catalog` image accessible. 1. Mirror the `ui-plugin-catalog` image to a private registry: ```bash - export REGISTRY_ENDPOINT="my-private-registry.com" - docker pull rancher/ui-plugin-catalog:1.0.0 - docker tag rancher/ui-plugin-catalog:1.0.0 $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog:1.0.0 - docker push $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog:1.0.0 + export REGISTRY_ENDPOINT= # e.g. "my-private-registry.com" + docker pull rancher/ui-plugin-catalog: + docker tag rancher/ui-plugin-catalog: $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog: + docker push $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog: 2. Use the mirrored image to create a Kubernetes [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/): ```yaml apiVersion: apps/v1 @@ -139,10 +139,10 @@ Rancher provides some extensions, such as Kubewarden and Elemental, through the spec: containers: - name: server - image: my-private-registry.com/rancher/ui-plugin-catalog:1.0.0 + image: /rancher/ui-plugin-catalog: imagePullPolicy: Always imagePullSecrets: - - name: my-registry-credentials + - name: ``` 3. Expose the deployment by creating a [ClusterIP service](https://kubernetes.io/docs/concepts/services-networking/service/#type-clusterip): ```yaml @@ -176,7 +176,7 @@ After you successfully set up these resources, you can install the extensions fr ### Importing and Installing Extensions in an Air-gapped Environment 1. Find the address of the container image repository that you want to import as an extension. You should import and use the latest tagged version of the image to ensure you receive the latest features and security updates. - * **(Optional)** If the container image is private: [Create](../how-to-guides/new-user-guides/kubernetes-resources-setup/secrets.md) a registry secret within the `cattle-ui-plugin-system` namespace. Enter the domain of the image address in the **Registry Domain Name** field. + - **(Optional)** If the container image is private: [Create](../how-to-guides/new-user-guides/kubernetes-resources-setup/secrets.md) a registry secret within the `cattle-ui-plugin-system` namespace. Enter the domain of the image address in the **Registry Domain Name** field. 1. Click **☰**, then select **Extensions**, under **Configuration**. 1. On the top right, click **⋮ > Manage Extension Catalogs**. 1. Select the **Import Extension Catalog** button. @@ -184,8 +184,8 @@ After you successfully set up these resources, you can install the extensions fr * **(Optional)** If the container image is private, select the secret you just created from the **Pull Secrets** drop-down menu. 1. Click **Load**. The extension will now be **Pending**. 1. Return to the **Extensions** page. -1. Select the **Available** tab, and click the **Reload** button to make sure that the list of extensions is up to date. -1. Find the extension you just added, and click the **Install** button. +1. Select the **Available** tab, and click **Reload** to make sure that the list of extensions is up to date. +1. Find the extension you just added, and click **Install**. ### Updating and Upgrading an Extensions Repository in an Air-gapped Environment From ab25aa2067cfb9b0902181e7cdbbfd5a6aace845 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Fri, 5 Apr 2024 13:41:47 -0700 Subject: [PATCH 10/67] Updating CAPI term use for better readability and applying review changes to add numbered ordering. Signed-off-by: Sunil Singh --- .../cluster-api/overview.md | 62 +++++++++---------- .../cluster-api/overview.md | 62 +++++++++---------- .../cluster-api/overview.md | 62 +++++++++---------- 3 files changed, 93 insertions(+), 93 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 2e050ee1804..ad587083b28 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -15,7 +15,7 @@ title: Overview You can install the Rancher Turtles operator via the Rancher UI, or with Helm. The first method is recommended for most environments. :::caution -If you already have the Cluster API Operator installed in your cluster, you must use the [manual helm installation method](./overview.md#via-helm-install). +If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#via-helm-install). ::: :::info prerequisite @@ -28,21 +28,21 @@ To simplify setting up Rancher for installing Rancher Turtles, the official Ranc ### Via Rancher UI -By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the Cluster API Extension. +By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the CAPI Extension. #### Installation -- Click **☰**. Under **Explore Cluster** in the left navigation menu, select **local**. -- In the left navigation menu of the **Cluster Dashboard**, select **Apps > Repositories**. -- Click **Create** to add a new repository. -- Enter the following: - - **Name**: turtles - - **Index URL**: https://rancher.github.io/turtles -- Wait until the new repository has a status of `Active`. -- In the left navigation menu, select **Apps > Charts**. -- Enter "turtles" into the search filter to find the Turtles chart. -- Click **Rancher Turtles - the Cluster API Extension**. -- Click **Install > Next > Install**. +1. Click **☰**. Under **Explore Cluster** in the left navigation menu, select **local**. +1. In the left navigation menu of the **Cluster Dashboard**, select **Apps > Repositories**. +1. Click **Create** to add a new repository. +1. Enter the following: + - **Name**: turtles + - **Index URL**: https://rancher.github.io/turtles +1. Wait until the new repository has a status of `Active`. +1. In the left navigation menu, select **Apps > Charts**. +1. Enter "turtles" into the search filter to find the Turtles chart. +1. Click **Rancher Turtles - the Cluster API Extension**. +1. Click **Install > Next > Install**. This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#via-helm-install). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). @@ -53,24 +53,24 @@ The installation may take a few minutes and once completed you will see the foll ### Via Helm Install -The Helm installation options for Rancher Turtles has two paths depending on CAPI operator use: -- [Install Rancher Turtles with Cluster API Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency). -- [Install Rancher Turtles without Cluster API Operator](#install-rancher-turtles-without-cluster-api-operator-as-a-helm-dependency). +There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: +- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-capi-as-a-helm-dependency). +- [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-operator-capi-as-a-helm-dependency). -The Cluster API Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. +The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. -CAPI Operator allows for handling the lifecycle of Cluster API providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). +The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). -#### Install Rancher Turtles with `Cluster API Operator` as a Helm dependency +#### Install Rancher Turtles with `Cluster API Operator (CAPI)` as a Helm dependency -The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: +1. The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: ```bash helm repo add turtles https://rancher.github.io/turtles helm repo update ``` -As mentioned before, installing Rancher Turtles requires the [Cluster API Operator](https://github.com/kubernetes-sigs/cluster-api-operator). The Helm chart can automatically install it with a minimal set of flags: +2. As mentioned before, installing Rancher Turtles requires the [CAPI Operator](https://github.com/kubernetes-sigs/cluster-api-operator). The Helm chart can automatically install it with a minimal set of flags: ```bash helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ @@ -80,17 +80,17 @@ helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ --timeout 180s ``` -This operation could take a few minutes and once completed you can review the installed controllers listed below: +3. This operation could take a few minutes and once completed you can review the installed controllers listed below: - `rancher-turtles-controller` - `capi-operator` :::note -- If `cert-manager` is already available in the cluster, to avoid conflicts, disable its installation as a Rancher Turtles dependency: +- If `cert-manager` is already available in the cluster, disable its installation as a Rancher Turtles dependency. This prevents dependency conflicts: `--set cluster-api-operator.cert-manager.enabled=false` - For a list of Rancher Turtles versions, refer to the [Turtles release page](https://github.com/rancher/turtles/releases). ::: -This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional Cluster API functionality. +This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional CAPI functionality. If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including enabling the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL`. @@ -120,20 +120,20 @@ stringData: For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) ::: -#### Install Rancher Turtles without `Cluster API Operator` as a Helm dependency +#### Install Rancher Turtles without `Cluster API Operator (CAPI)` as a Helm dependency :::note -Remember that if you opt for this installation option, you must manage the Cluster API Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. +Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. ::: -Add the `rancher-turtles` chart from the in https://rancher.github.io/turtles Helm repository before installation: +1. Add the `rancher-turtles` chart from the in https://rancher.github.io/turtles Helm repository before installation: ```bash helm repo add turtles https://rancher.github.io/turtles helm repo update ``` -Install the chart into the `rancher-turtles-system` namespace: +2. Install the chart into the `rancher-turtles-system` namespace: ```bash helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 @@ -146,13 +146,13 @@ helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 The previous commands tell Helm to ignore installing `cluster-api-operator` as a dependency. -This operation could take a few minutes and once completed you can review the installed controller listed below: +3. This operation could take a few minutes and once completed you can review the installed controller listed below: - `rancher-turtles-controller` ## Uninstalling Rancher Turtles :::caution -When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the Cluster API Operator cleanup. This includes cleaning up Cluster API Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. +When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the CAPI Operator cleanup. This includes cleaning up CAPI Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. To simplify uninstalling Rancher Turtles (via Rancher Manager or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. @@ -189,7 +189,7 @@ kubectl apply -f feature.yaml ## Architecture Diagram -Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the Cluster API (CAPI) for cluster management. +Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the CAPI operator for cluster management. ![overview](/img/30000ft_view.png) diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index 2e050ee1804..ad587083b28 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -15,7 +15,7 @@ title: Overview You can install the Rancher Turtles operator via the Rancher UI, or with Helm. The first method is recommended for most environments. :::caution -If you already have the Cluster API Operator installed in your cluster, you must use the [manual helm installation method](./overview.md#via-helm-install). +If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#via-helm-install). ::: :::info prerequisite @@ -28,21 +28,21 @@ To simplify setting up Rancher for installing Rancher Turtles, the official Ranc ### Via Rancher UI -By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the Cluster API Extension. +By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the CAPI Extension. #### Installation -- Click **☰**. Under **Explore Cluster** in the left navigation menu, select **local**. -- In the left navigation menu of the **Cluster Dashboard**, select **Apps > Repositories**. -- Click **Create** to add a new repository. -- Enter the following: - - **Name**: turtles - - **Index URL**: https://rancher.github.io/turtles -- Wait until the new repository has a status of `Active`. -- In the left navigation menu, select **Apps > Charts**. -- Enter "turtles" into the search filter to find the Turtles chart. -- Click **Rancher Turtles - the Cluster API Extension**. -- Click **Install > Next > Install**. +1. Click **☰**. Under **Explore Cluster** in the left navigation menu, select **local**. +1. In the left navigation menu of the **Cluster Dashboard**, select **Apps > Repositories**. +1. Click **Create** to add a new repository. +1. Enter the following: + - **Name**: turtles + - **Index URL**: https://rancher.github.io/turtles +1. Wait until the new repository has a status of `Active`. +1. In the left navigation menu, select **Apps > Charts**. +1. Enter "turtles" into the search filter to find the Turtles chart. +1. Click **Rancher Turtles - the Cluster API Extension**. +1. Click **Install > Next > Install**. This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#via-helm-install). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). @@ -53,24 +53,24 @@ The installation may take a few minutes and once completed you will see the foll ### Via Helm Install -The Helm installation options for Rancher Turtles has two paths depending on CAPI operator use: -- [Install Rancher Turtles with Cluster API Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency). -- [Install Rancher Turtles without Cluster API Operator](#install-rancher-turtles-without-cluster-api-operator-as-a-helm-dependency). +There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: +- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-capi-as-a-helm-dependency). +- [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-operator-capi-as-a-helm-dependency). -The Cluster API Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. +The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. -CAPI Operator allows for handling the lifecycle of Cluster API providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). +The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). -#### Install Rancher Turtles with `Cluster API Operator` as a Helm dependency +#### Install Rancher Turtles with `Cluster API Operator (CAPI)` as a Helm dependency -The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: +1. The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: ```bash helm repo add turtles https://rancher.github.io/turtles helm repo update ``` -As mentioned before, installing Rancher Turtles requires the [Cluster API Operator](https://github.com/kubernetes-sigs/cluster-api-operator). The Helm chart can automatically install it with a minimal set of flags: +2. As mentioned before, installing Rancher Turtles requires the [CAPI Operator](https://github.com/kubernetes-sigs/cluster-api-operator). The Helm chart can automatically install it with a minimal set of flags: ```bash helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ @@ -80,17 +80,17 @@ helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ --timeout 180s ``` -This operation could take a few minutes and once completed you can review the installed controllers listed below: +3. This operation could take a few minutes and once completed you can review the installed controllers listed below: - `rancher-turtles-controller` - `capi-operator` :::note -- If `cert-manager` is already available in the cluster, to avoid conflicts, disable its installation as a Rancher Turtles dependency: +- If `cert-manager` is already available in the cluster, disable its installation as a Rancher Turtles dependency. This prevents dependency conflicts: `--set cluster-api-operator.cert-manager.enabled=false` - For a list of Rancher Turtles versions, refer to the [Turtles release page](https://github.com/rancher/turtles/releases). ::: -This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional Cluster API functionality. +This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional CAPI functionality. If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including enabling the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL`. @@ -120,20 +120,20 @@ stringData: For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) ::: -#### Install Rancher Turtles without `Cluster API Operator` as a Helm dependency +#### Install Rancher Turtles without `Cluster API Operator (CAPI)` as a Helm dependency :::note -Remember that if you opt for this installation option, you must manage the Cluster API Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. +Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. ::: -Add the `rancher-turtles` chart from the in https://rancher.github.io/turtles Helm repository before installation: +1. Add the `rancher-turtles` chart from the in https://rancher.github.io/turtles Helm repository before installation: ```bash helm repo add turtles https://rancher.github.io/turtles helm repo update ``` -Install the chart into the `rancher-turtles-system` namespace: +2. Install the chart into the `rancher-turtles-system` namespace: ```bash helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 @@ -146,13 +146,13 @@ helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 The previous commands tell Helm to ignore installing `cluster-api-operator` as a dependency. -This operation could take a few minutes and once completed you can review the installed controller listed below: +3. This operation could take a few minutes and once completed you can review the installed controller listed below: - `rancher-turtles-controller` ## Uninstalling Rancher Turtles :::caution -When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the Cluster API Operator cleanup. This includes cleaning up Cluster API Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. +When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the CAPI Operator cleanup. This includes cleaning up CAPI Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. To simplify uninstalling Rancher Turtles (via Rancher Manager or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. @@ -189,7 +189,7 @@ kubectl apply -f feature.yaml ## Architecture Diagram -Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the Cluster API (CAPI) for cluster management. +Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the CAPI operator for cluster management. ![overview](/img/30000ft_view.png) diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index 2e050ee1804..ad587083b28 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -15,7 +15,7 @@ title: Overview You can install the Rancher Turtles operator via the Rancher UI, or with Helm. The first method is recommended for most environments. :::caution -If you already have the Cluster API Operator installed in your cluster, you must use the [manual helm installation method](./overview.md#via-helm-install). +If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#via-helm-install). ::: :::info prerequisite @@ -28,21 +28,21 @@ To simplify setting up Rancher for installing Rancher Turtles, the official Ranc ### Via Rancher UI -By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the Cluster API Extension. +By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the CAPI Extension. #### Installation -- Click **☰**. Under **Explore Cluster** in the left navigation menu, select **local**. -- In the left navigation menu of the **Cluster Dashboard**, select **Apps > Repositories**. -- Click **Create** to add a new repository. -- Enter the following: - - **Name**: turtles - - **Index URL**: https://rancher.github.io/turtles -- Wait until the new repository has a status of `Active`. -- In the left navigation menu, select **Apps > Charts**. -- Enter "turtles" into the search filter to find the Turtles chart. -- Click **Rancher Turtles - the Cluster API Extension**. -- Click **Install > Next > Install**. +1. Click **☰**. Under **Explore Cluster** in the left navigation menu, select **local**. +1. In the left navigation menu of the **Cluster Dashboard**, select **Apps > Repositories**. +1. Click **Create** to add a new repository. +1. Enter the following: + - **Name**: turtles + - **Index URL**: https://rancher.github.io/turtles +1. Wait until the new repository has a status of `Active`. +1. In the left navigation menu, select **Apps > Charts**. +1. Enter "turtles" into the search filter to find the Turtles chart. +1. Click **Rancher Turtles - the Cluster API Extension**. +1. Click **Install > Next > Install**. This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#via-helm-install). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). @@ -53,24 +53,24 @@ The installation may take a few minutes and once completed you will see the foll ### Via Helm Install -The Helm installation options for Rancher Turtles has two paths depending on CAPI operator use: -- [Install Rancher Turtles with Cluster API Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency). -- [Install Rancher Turtles without Cluster API Operator](#install-rancher-turtles-without-cluster-api-operator-as-a-helm-dependency). +There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: +- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-capi-as-a-helm-dependency). +- [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-operator-capi-as-a-helm-dependency). -The Cluster API Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. +The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. -CAPI Operator allows for handling the lifecycle of Cluster API providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). +The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). -#### Install Rancher Turtles with `Cluster API Operator` as a Helm dependency +#### Install Rancher Turtles with `Cluster API Operator (CAPI)` as a Helm dependency -The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: +1. The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: ```bash helm repo add turtles https://rancher.github.io/turtles helm repo update ``` -As mentioned before, installing Rancher Turtles requires the [Cluster API Operator](https://github.com/kubernetes-sigs/cluster-api-operator). The Helm chart can automatically install it with a minimal set of flags: +2. As mentioned before, installing Rancher Turtles requires the [CAPI Operator](https://github.com/kubernetes-sigs/cluster-api-operator). The Helm chart can automatically install it with a minimal set of flags: ```bash helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ @@ -80,17 +80,17 @@ helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ --timeout 180s ``` -This operation could take a few minutes and once completed you can review the installed controllers listed below: +3. This operation could take a few minutes and once completed you can review the installed controllers listed below: - `rancher-turtles-controller` - `capi-operator` :::note -- If `cert-manager` is already available in the cluster, to avoid conflicts, disable its installation as a Rancher Turtles dependency: +- If `cert-manager` is already available in the cluster, disable its installation as a Rancher Turtles dependency. This prevents dependency conflicts: `--set cluster-api-operator.cert-manager.enabled=false` - For a list of Rancher Turtles versions, refer to the [Turtles release page](https://github.com/rancher/turtles/releases). ::: -This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional Cluster API functionality. +This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional CAPI functionality. If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including enabling the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL`. @@ -120,20 +120,20 @@ stringData: For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) ::: -#### Install Rancher Turtles without `Cluster API Operator` as a Helm dependency +#### Install Rancher Turtles without `Cluster API Operator (CAPI)` as a Helm dependency :::note -Remember that if you opt for this installation option, you must manage the Cluster API Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. +Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. ::: -Add the `rancher-turtles` chart from the in https://rancher.github.io/turtles Helm repository before installation: +1. Add the `rancher-turtles` chart from the in https://rancher.github.io/turtles Helm repository before installation: ```bash helm repo add turtles https://rancher.github.io/turtles helm repo update ``` -Install the chart into the `rancher-turtles-system` namespace: +2. Install the chart into the `rancher-turtles-system` namespace: ```bash helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 @@ -146,13 +146,13 @@ helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 The previous commands tell Helm to ignore installing `cluster-api-operator` as a dependency. -This operation could take a few minutes and once completed you can review the installed controller listed below: +3. This operation could take a few minutes and once completed you can review the installed controller listed below: - `rancher-turtles-controller` ## Uninstalling Rancher Turtles :::caution -When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the Cluster API Operator cleanup. This includes cleaning up Cluster API Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. +When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the CAPI Operator cleanup. This includes cleaning up CAPI Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. To simplify uninstalling Rancher Turtles (via Rancher Manager or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. @@ -189,7 +189,7 @@ kubectl apply -f feature.yaml ## Architecture Diagram -Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the Cluster API (CAPI) for cluster management. +Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the CAPI operator for cluster management. ![overview](/img/30000ft_view.png) From eb5b3c8b10ebbaf2ed176615708e6ff1f37631cb Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Fri, 5 Apr 2024 13:48:23 -0700 Subject: [PATCH 11/67] Updating title for search term CAPI and Cluster API. Signed-off-by: Sunil Singh --- docs/integrations-in-rancher/cluster-api/overview.md | 8 ++++---- .../integrations-in-rancher/cluster-api/overview.md | 8 ++++---- .../integrations-in-rancher/cluster-api/overview.md | 8 ++++---- 3 files changed, 12 insertions(+), 12 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index ad587083b28..c8105737d4e 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -54,14 +54,14 @@ The installation may take a few minutes and once completed you will see the foll ### Via Helm Install There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: -- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-capi-as-a-helm-dependency). -- [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-operator-capi-as-a-helm-dependency). +- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). +- [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). -#### Install Rancher Turtles with `Cluster API Operator (CAPI)` as a Helm dependency +#### Install Rancher Turtles with `Cluster API (CAPI) Operator` as a Helm dependency 1. The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: @@ -120,7 +120,7 @@ stringData: For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) ::: -#### Install Rancher Turtles without `Cluster API Operator (CAPI)` as a Helm dependency +#### Install Rancher Turtles without `Cluster API (CAPI) Operator` as a Helm dependency :::note Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index ad587083b28..c8105737d4e 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -54,14 +54,14 @@ The installation may take a few minutes and once completed you will see the foll ### Via Helm Install There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: -- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-capi-as-a-helm-dependency). -- [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-operator-capi-as-a-helm-dependency). +- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). +- [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). -#### Install Rancher Turtles with `Cluster API Operator (CAPI)` as a Helm dependency +#### Install Rancher Turtles with `Cluster API (CAPI) Operator` as a Helm dependency 1. The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: @@ -120,7 +120,7 @@ stringData: For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) ::: -#### Install Rancher Turtles without `Cluster API Operator (CAPI)` as a Helm dependency +#### Install Rancher Turtles without `Cluster API (CAPI) Operator` as a Helm dependency :::note Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index ad587083b28..159d232e07c 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -54,14 +54,14 @@ The installation may take a few minutes and once completed you will see the foll ### Via Helm Install There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: -- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-operator-capi-as-a-helm-dependency). -- [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-operator-capi-as-a-helm-dependency). +- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-capi-as-a-helm-dependency). +- [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). -#### Install Rancher Turtles with `Cluster API Operator (CAPI)` as a Helm dependency +#### Install Rancher Turtles with `Cluster API (CAPI) Operator` as a Helm dependency 1. The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: @@ -120,7 +120,7 @@ stringData: For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) ::: -#### Install Rancher Turtles without `Cluster API Operator (CAPI)` as a Helm dependency +#### Install Rancher Turtles without `Cluster API (CAPI) Operator` as a Helm dependency :::note Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. From 6f1a4ef1bf003ab74cf809b5b5fe7c563ff48f41 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Fri, 5 Apr 2024 13:49:37 -0700 Subject: [PATCH 12/67] Updating broken link. Signed-off-by: Sunil Singh --- docs/integrations-in-rancher/cluster-api/overview.md | 2 +- .../version-2.7/integrations-in-rancher/cluster-api/overview.md | 2 +- .../version-2.8/integrations-in-rancher/cluster-api/overview.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index c8105737d4e..0e6e72f994d 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -57,7 +57,7 @@ There are two ways to install Rancher Turtles with Helm, depending on whether yo - [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). - [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). -The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. +The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index c8105737d4e..0e6e72f994d 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -57,7 +57,7 @@ There are two ways to install Rancher Turtles with Helm, depending on whether yo - [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). - [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). -The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. +The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index 159d232e07c..ac4c71d3033 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -57,7 +57,7 @@ There are two ways to install Rancher Turtles with Helm, depending on whether yo - [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-capi-as-a-helm-dependency). - [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). -The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. +The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). From aff1de294b7156a7c5d69eaedda19b898612b871 Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Mon, 8 Apr 2024 11:25:21 -0400 Subject: [PATCH 13/67] #1213 Instructions on altering minimum password length (#1219) * 1213 Instructions on altering minimum password length * typo fix plus note about versions --- .../authentication-config/manage-users-and-groups.md | 8 ++++++++ .../authentication-config/manage-users-and-groups.md | 8 ++++++++ .../authentication-config/manage-users-and-groups.md | 8 ++++++++ .../authentication-config/manage-users-and-groups.md | 8 ++++++++ 4 files changed, 32 insertions(+) diff --git a/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md b/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md index ec39be1f01b..dcfcacda3c3 100644 --- a/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md +++ b/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md @@ -70,6 +70,14 @@ Since SAML does not support user lookup, SAML-based authentication providers do ::: +## Minimum Password Length + +By default, user passwords must be at least 12 characters long. However, you can customize the password length requirement: + +1. In the upper left corner, click **☰ > Global Settings**. +1. Go to **`password-min-length`** and click **⋮ > Edit Setting**. +1. Enter an integer value between 2 and 256, and click **Save**. + ## Session Length The default length (TTL) of each user session is adjustable. The default session length is 16 hours. diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md index ec39be1f01b..030f7e36057 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md @@ -70,6 +70,14 @@ Since SAML does not support user lookup, SAML-based authentication providers do ::: +## Minimum Password Length + +By default, user passwords must be at least 12 characters long. Starting in Rancher v2.6.9, you can customize the password length requirement: + +1. In the upper left corner, click **☰ > Global Settings**. +1. Go to **`password-min-length`** and click **⋮ > Edit Setting**. +1. Enter an integer value between 2 and 256, and click **Save**. + ## Session Length The default length (TTL) of each user session is adjustable. The default session length is 16 hours. diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md index ec39be1f01b..dcfcacda3c3 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md @@ -70,6 +70,14 @@ Since SAML does not support user lookup, SAML-based authentication providers do ::: +## Minimum Password Length + +By default, user passwords must be at least 12 characters long. However, you can customize the password length requirement: + +1. In the upper left corner, click **☰ > Global Settings**. +1. Go to **`password-min-length`** and click **⋮ > Edit Setting**. +1. Enter an integer value between 2 and 256, and click **Save**. + ## Session Length The default length (TTL) of each user session is adjustable. The default session length is 16 hours. diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md index ec39be1f01b..dcfcacda3c3 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md @@ -70,6 +70,14 @@ Since SAML does not support user lookup, SAML-based authentication providers do ::: +## Minimum Password Length + +By default, user passwords must be at least 12 characters long. However, you can customize the password length requirement: + +1. In the upper left corner, click **☰ > Global Settings**. +1. Go to **`password-min-length`** and click **⋮ > Edit Setting**. +1. Enter an integer value between 2 and 256, and click **Save**. + ## Session Length The default length (TTL) of each user session is adjustable. The default session length is 16 hours. From beeb4cf485bf81cf7002dc9250ac8672f4dde583 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 8 Apr 2024 08:52:01 -0700 Subject: [PATCH 14/67] Adjusting the CAPI page to include a link to the CAPI site for unfamiliar users, and moved the architecture section in the overview as the first item. Signed-off-by: Sunil Singh --- docs/integrations-in-rancher/cluster-api/capi.md | 4 ++-- .../cluster-api/overview.md | 14 ++++++++------ 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/capi.md b/docs/integrations-in-rancher/cluster-api/capi.md index 68d2e67665b..6c66d0a5bf6 100644 --- a/docs/integrations-in-rancher/cluster-api/capi.md +++ b/docs/integrations-in-rancher/cluster-api/capi.md @@ -1,12 +1,12 @@ --- -title: Cluster API with Rancher Turtles +title: Cluster API (CAPI) with Rancher Turtles --- -Rancher Turtles is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: +[Rancher Turtles](https://turtles.docs.rancher.com/) is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: - Import CAPI clusters into Rancher, by installing the Rancher Cluster Agent in CAPI provisioned clusters. - Configure the CAPI Operator. diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 0e6e72f994d..613f91d572d 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -6,6 +6,12 @@ title: Overview +## Architecture Diagram + +Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the CAPI operator for cluster management. + +![overview](/img/30000ft_view.png) + ## Demo @@ -15,7 +21,9 @@ title: Overview You can install the Rancher Turtles operator via the Rancher UI, or with Helm. The first method is recommended for most environments. :::caution + If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#via-helm-install). + ::: :::info prerequisite @@ -187,12 +195,6 @@ spec: kubectl apply -f feature.yaml ``` -## Architecture Diagram - -Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the CAPI operator for cluster management. - -![overview](/img/30000ft_view.png) - ## Security [SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. From 7f4f0f2b94146f85cfec389c89529602da573675 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 8 Apr 2024 09:01:40 -0700 Subject: [PATCH 15/67] Moving the demo to after the text instructions, demo includes adding Rancher Turtles via UI, creating/importing a CAPI cluster using Fleet GitOps, and installing monitoring on the CAPI clusters so I will keep it as its own section. Also some syntax adjustments. Signed-off-by: Sunil Singh --- .../cluster-api/overview.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 613f91d572d..65648fa87f0 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -8,7 +8,7 @@ title: Overview ## Architecture Diagram -Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the CAPI operator for cluster management. +Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the CAPI operator for cluster management. ![overview](/img/30000ft_view.png) @@ -125,7 +125,9 @@ stringData: ``` :::info + For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) + ::: #### Install Rancher Turtles without `Cluster API (CAPI) Operator` as a Helm dependency @@ -162,7 +164,7 @@ The previous commands tell Helm to ignore installing `cluster-api-operator` as a :::caution When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the CAPI Operator cleanup. This includes cleaning up CAPI Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. -To simplify uninstalling Rancher Turtles (via Rancher Manager or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: +To simplify uninstalling Rancher Turtles (via Rancher or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. - Deletes the CAPI `deployments` that are no longer needed. ::: @@ -176,10 +178,12 @@ helm uninstall -n rancher-turtles-system rancher-turtles --cascade foreground -- This may take a few minutes to complete. :::note + Remember that, if you use a different name for the installation or a different namespace, you may need to customize the command for your specific configuration. + ::: -Once Rancher Turtles is uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: +After Rancher Turtles is uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: 1. Create a `feature.yaml` file, with `embedded-cluster-api` set to true: ```yaml title="feature.yaml" @@ -195,6 +199,10 @@ spec: kubectl apply -f feature.yaml ``` +## Demo + + + ## Security [SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. From 349219413330dc56c2ba66f7af790589e7c0cc00 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 8 Apr 2024 09:05:32 -0700 Subject: [PATCH 16/67] Removing the old demo section and moving the prereqs to the beginning of the page as a top level section. Signed-off-by: Sunil Singh --- docs/integrations-in-rancher/cluster-api/overview.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 65648fa87f0..6a04d36de02 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -12,9 +12,14 @@ Below is a visual representation of the key components of Rancher Turtles and th ![overview](/img/30000ft_view.png) -## Demo +## Prerequisites - +Before installing Rancher Turtles in your Rancher environment, you must disable Rancher's `embedded-cluster-api` functionality. This also includes cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. + +To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that removes the following: + +- Disables the `embedded-cluster-api` feature in Rancher. +- Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. ## Installing the Rancher Turtles Operator From c4502d7557a140ac489779517a3e3b82642487ae Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 8 Apr 2024 09:24:04 -0700 Subject: [PATCH 17/67] Adjusting phrasing and some syntax after review. Signed-off-by: Sunil Singh --- .../cluster-api/capi.md | 2 +- .../cluster-api/overview.md | 39 ++++++++++++------- 2 files changed, 26 insertions(+), 15 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/capi.md b/docs/integrations-in-rancher/cluster-api/capi.md index 6c66d0a5bf6..7cb1dae14e6 100644 --- a/docs/integrations-in-rancher/cluster-api/capi.md +++ b/docs/integrations-in-rancher/cluster-api/capi.md @@ -9,6 +9,6 @@ title: Cluster API (CAPI) with Rancher Turtles [Rancher Turtles](https://turtles.docs.rancher.com/) is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: - Import CAPI clusters into Rancher, by installing the Rancher Cluster Agent in CAPI provisioned clusters. -- Configure the CAPI Operator. +- Configure the [CAPI Operator](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values). The [Overview](./overview.md) section outlines installation options, Rancher Turtles architecture, and a brief demo. For more details, see the [Rancher Turtles documentation](https://turtles.docs.rancher.com/). diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 6a04d36de02..c15753877cc 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -39,12 +39,10 @@ To simplify setting up Rancher for installing Rancher Turtles, the official Ranc - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. ::: -### Via Rancher UI +### Installing via the Rancher UI By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the CAPI Extension. -#### Installation - 1. Click **☰**. Under **Explore Cluster** in the left navigation menu, select **local**. 1. In the left navigation menu of the **Cluster Dashboard**, select **Apps > Repositories**. 1. Click **Create** to add a new repository. @@ -59,12 +57,13 @@ By adding the Turtles repository via the Rancher UI, Rancher can process the ins This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#via-helm-install). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). -The installation may take a few minutes and once completed you will see the following new deployments in the cluster: +The installation may take a few minutes and after completing you can see the following new deployments in the cluster: + - `rancher-turtles-system/rancher-turtles-controller-manager` - `rancher-turtles-system/rancher-turtles-cluster-api-operator` - `capi-system/capi-controller-manager` -### Via Helm Install +### Installing via Helm There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: - [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). @@ -74,9 +73,9 @@ The CAPI Operator is required for installing Rancher Turtles. You can choose whe The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). -#### Install Rancher Turtles with `Cluster API (CAPI) Operator` as a Helm dependency +#### Installing Rancher Turtles with `Cluster API (CAPI) Operator` as a Helm dependency -1. The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: +1. Add the Helm repository containing the `rancher-turtles` chart as the first step in installation: ```bash helm repo add turtles https://rancher.github.io/turtles @@ -86,26 +85,29 @@ helm repo update 2. As mentioned before, installing Rancher Turtles requires the [CAPI Operator](https://github.com/kubernetes-sigs/cluster-api-operator). The Helm chart can automatically install it with a minimal set of flags: ```bash -helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ +helm install rancher-turtles turtles/rancher-turtles --version \ -n rancher-turtles-system \ --dependency-update \ --create-namespace --wait \ --timeout 180s ``` -3. This operation could take a few minutes and once completed you can review the installed controllers listed below: +3. This operation could take a few minutes and after completing you can review the installed controllers listed below: + - `rancher-turtles-controller` - `capi-operator` :::note + - If `cert-manager` is already available in the cluster, disable its installation as a Rancher Turtles dependency. This prevents dependency conflicts: `--set cluster-api-operator.cert-manager.enabled=false` - For a list of Rancher Turtles versions, refer to the [Turtles release page](https://github.com/rancher/turtles/releases). + ::: This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional CAPI functionality. -If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including enabling the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL`. +If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name Helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including enabling the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL`. ```bash helm install ... @@ -135,13 +137,15 @@ For detailed information on the values supported by the chart and their usage, r ::: -#### Install Rancher Turtles without `Cluster API (CAPI) Operator` as a Helm dependency +#### Installing Rancher Turtles without `Cluster API (CAPI) Operator` as a Helm dependency :::note + Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. + ::: -1. Add the `rancher-turtles` chart from the in https://rancher.github.io/turtles Helm repository before installation: +1. Add the Helm repository containing the `rancher-turtles` chart as the first step in installation: ```bash helm repo add turtles https://rancher.github.io/turtles @@ -151,7 +155,7 @@ helm repo update 2. Install the chart into the `rancher-turtles-system` namespace: ```bash -helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 +helm install rancher-turtles turtles/rancher-turtles --version -n rancher-turtles-system --set cluster-api-operator.enabled=false --set cluster-api-operator.cluster-api.enabled=false @@ -161,17 +165,21 @@ helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 The previous commands tell Helm to ignore installing `cluster-api-operator` as a dependency. -3. This operation could take a few minutes and once completed you can review the installed controller listed below: +3. This operation could take a few minutes and after completing you can review the installed controller listed below: + - `rancher-turtles-controller` ## Uninstalling Rancher Turtles :::caution + When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the CAPI Operator cleanup. This includes cleaning up CAPI Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. To simplify uninstalling Rancher Turtles (via Rancher or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: + - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. - Deletes the CAPI `deployments` that are no longer needed. + ::: To uninstall Rancher Turtles: @@ -191,6 +199,7 @@ Remember that, if you use a different name for the installation or a different n After Rancher Turtles is uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: 1. Create a `feature.yaml` file, with `embedded-cluster-api` set to true: + ```yaml title="feature.yaml" apiVersion: management.cattle.io/v3 kind: Feature @@ -199,7 +208,9 @@ metadata: spec: value: true ``` + 2. Use `kubectl` to apply the `feature.yaml` file to the cluster: + ```bash kubectl apply -f feature.yaml ``` From c17765a0b59ceb8f221d618f53813d15937df4a7 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 8 Apr 2024 09:25:27 -0700 Subject: [PATCH 18/67] Updating the sidebars across versions to match page title. Signed-off-by: Sunil Singh --- sidebars.js | 2 +- versioned_sidebars/version-2.7-sidebars.json | 2 +- versioned_sidebars/version-2.8-sidebars.json | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/sidebars.js b/sidebars.js index ab01993973a..ac4e0200b3d 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1217,7 +1217,7 @@ const sidebars = { }, { "type": "category", - "label": "Cluster API with Rancher Turtles", + "label": "Cluster API (CAPI) with Rancher Turtles", "link": { "type": "doc", "id": "integrations-in-rancher/cluster-api/capi" diff --git a/versioned_sidebars/version-2.7-sidebars.json b/versioned_sidebars/version-2.7-sidebars.json index edb87c80c6f..5c8ee7ab98b 100644 --- a/versioned_sidebars/version-2.7-sidebars.json +++ b/versioned_sidebars/version-2.7-sidebars.json @@ -1217,7 +1217,7 @@ "integrations-in-rancher/longhorn", { "type": "category", - "label": "Cluster API with Rancher Turtles", + "label": "Cluster API (CAPI) with Rancher Turtles", "link": { "type": "doc", "id": "integrations-in-rancher/cluster-api/capi" diff --git a/versioned_sidebars/version-2.8-sidebars.json b/versioned_sidebars/version-2.8-sidebars.json index 2dc5f196a5e..824b2892428 100644 --- a/versioned_sidebars/version-2.8-sidebars.json +++ b/versioned_sidebars/version-2.8-sidebars.json @@ -1177,7 +1177,7 @@ }, { "type": "category", - "label": "Cluster API with Rancher Turtles", + "label": "Cluster API (CAPI) with Rancher Turtles", "link": { "type": "doc", "id": "integrations-in-rancher/cluster-api/capi" From 2056cce40133aa2bf7213728dc8449ee312cc3d1 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 8 Apr 2024 09:28:31 -0700 Subject: [PATCH 19/67] Syncing versions after review updates applied. Signed-off-by: Sunil Singh --- .../cluster-api/capi.md | 6 +- .../cluster-api/overview.md | 70 ++++++++++++------ .../cluster-api/capi.md | 6 +- .../cluster-api/overview.md | 72 +++++++++++++------ 4 files changed, 103 insertions(+), 51 deletions(-) diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/capi.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/capi.md index 68d2e67665b..7cb1dae14e6 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/capi.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/capi.md @@ -1,14 +1,14 @@ --- -title: Cluster API with Rancher Turtles +title: Cluster API (CAPI) with Rancher Turtles --- -Rancher Turtles is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: +[Rancher Turtles](https://turtles.docs.rancher.com/) is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: - Import CAPI clusters into Rancher, by installing the Rancher Cluster Agent in CAPI provisioned clusters. -- Configure the CAPI Operator. +- Configure the [CAPI Operator](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values). The [Overview](./overview.md) section outlines installation options, Rancher Turtles architecture, and a brief demo. For more details, see the [Rancher Turtles documentation](https://turtles.docs.rancher.com/). diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index 0e6e72f994d..c15753877cc 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -6,16 +6,29 @@ title: Overview -## Demo +## Architecture Diagram - +Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the CAPI operator for cluster management. + +![overview](/img/30000ft_view.png) + +## Prerequisites + +Before installing Rancher Turtles in your Rancher environment, you must disable Rancher's `embedded-cluster-api` functionality. This also includes cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. + +To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that removes the following: + +- Disables the `embedded-cluster-api` feature in Rancher. +- Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. ## Installing the Rancher Turtles Operator You can install the Rancher Turtles operator via the Rancher UI, or with Helm. The first method is recommended for most environments. :::caution + If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#via-helm-install). + ::: :::info prerequisite @@ -26,12 +39,10 @@ To simplify setting up Rancher for installing Rancher Turtles, the official Ranc - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. ::: -### Via Rancher UI +### Installing via the Rancher UI By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the CAPI Extension. -#### Installation - 1. Click **☰**. Under **Explore Cluster** in the left navigation menu, select **local**. 1. In the left navigation menu of the **Cluster Dashboard**, select **Apps > Repositories**. 1. Click **Create** to add a new repository. @@ -46,12 +57,13 @@ By adding the Turtles repository via the Rancher UI, Rancher can process the ins This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#via-helm-install). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). -The installation may take a few minutes and once completed you will see the following new deployments in the cluster: +The installation may take a few minutes and after completing you can see the following new deployments in the cluster: + - `rancher-turtles-system/rancher-turtles-controller-manager` - `rancher-turtles-system/rancher-turtles-cluster-api-operator` - `capi-system/capi-controller-manager` -### Via Helm Install +### Installing via Helm There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: - [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). @@ -61,9 +73,9 @@ The CAPI Operator is required for installing Rancher Turtles. You can choose whe The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). -#### Install Rancher Turtles with `Cluster API (CAPI) Operator` as a Helm dependency +#### Installing Rancher Turtles with `Cluster API (CAPI) Operator` as a Helm dependency -1. The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: +1. Add the Helm repository containing the `rancher-turtles` chart as the first step in installation: ```bash helm repo add turtles https://rancher.github.io/turtles @@ -73,26 +85,29 @@ helm repo update 2. As mentioned before, installing Rancher Turtles requires the [CAPI Operator](https://github.com/kubernetes-sigs/cluster-api-operator). The Helm chart can automatically install it with a minimal set of flags: ```bash -helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ +helm install rancher-turtles turtles/rancher-turtles --version \ -n rancher-turtles-system \ --dependency-update \ --create-namespace --wait \ --timeout 180s ``` -3. This operation could take a few minutes and once completed you can review the installed controllers listed below: +3. This operation could take a few minutes and after completing you can review the installed controllers listed below: + - `rancher-turtles-controller` - `capi-operator` :::note + - If `cert-manager` is already available in the cluster, disable its installation as a Rancher Turtles dependency. This prevents dependency conflicts: `--set cluster-api-operator.cert-manager.enabled=false` - For a list of Rancher Turtles versions, refer to the [Turtles release page](https://github.com/rancher/turtles/releases). + ::: This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional CAPI functionality. -If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including enabling the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL`. +If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name Helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including enabling the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL`. ```bash helm install ... @@ -117,16 +132,20 @@ stringData: ``` :::info + For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) + ::: -#### Install Rancher Turtles without `Cluster API (CAPI) Operator` as a Helm dependency +#### Installing Rancher Turtles without `Cluster API (CAPI) Operator` as a Helm dependency :::note + Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. + ::: -1. Add the `rancher-turtles` chart from the in https://rancher.github.io/turtles Helm repository before installation: +1. Add the Helm repository containing the `rancher-turtles` chart as the first step in installation: ```bash helm repo add turtles https://rancher.github.io/turtles @@ -136,7 +155,7 @@ helm repo update 2. Install the chart into the `rancher-turtles-system` namespace: ```bash -helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 +helm install rancher-turtles turtles/rancher-turtles --version -n rancher-turtles-system --set cluster-api-operator.enabled=false --set cluster-api-operator.cluster-api.enabled=false @@ -146,17 +165,21 @@ helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 The previous commands tell Helm to ignore installing `cluster-api-operator` as a dependency. -3. This operation could take a few minutes and once completed you can review the installed controller listed below: +3. This operation could take a few minutes and after completing you can review the installed controller listed below: + - `rancher-turtles-controller` ## Uninstalling Rancher Turtles :::caution + When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the CAPI Operator cleanup. This includes cleaning up CAPI Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. -To simplify uninstalling Rancher Turtles (via Rancher Manager or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: +To simplify uninstalling Rancher Turtles (via Rancher or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: + - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. - Deletes the CAPI `deployments` that are no longer needed. + ::: To uninstall Rancher Turtles: @@ -168,12 +191,15 @@ helm uninstall -n rancher-turtles-system rancher-turtles --cascade foreground -- This may take a few minutes to complete. :::note + Remember that, if you use a different name for the installation or a different namespace, you may need to customize the command for your specific configuration. + ::: -Once Rancher Turtles is uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: +After Rancher Turtles is uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: 1. Create a `feature.yaml` file, with `embedded-cluster-api` set to true: + ```yaml title="feature.yaml" apiVersion: management.cattle.io/v3 kind: Feature @@ -182,16 +208,16 @@ metadata: spec: value: true ``` + 2. Use `kubectl` to apply the `feature.yaml` file to the cluster: + ```bash kubectl apply -f feature.yaml ``` -## Architecture Diagram +## Demo -Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the CAPI operator for cluster management. - -![overview](/img/30000ft_view.png) + ## Security diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md index 68d2e67665b..7cb1dae14e6 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md @@ -1,14 +1,14 @@ --- -title: Cluster API with Rancher Turtles +title: Cluster API (CAPI) with Rancher Turtles --- -Rancher Turtles is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: +[Rancher Turtles](https://turtles.docs.rancher.com/) is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: - Import CAPI clusters into Rancher, by installing the Rancher Cluster Agent in CAPI provisioned clusters. -- Configure the CAPI Operator. +- Configure the [CAPI Operator](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values). The [Overview](./overview.md) section outlines installation options, Rancher Turtles architecture, and a brief demo. For more details, see the [Rancher Turtles documentation](https://turtles.docs.rancher.com/). diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index ac4c71d3033..c15753877cc 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -6,16 +6,29 @@ title: Overview -## Demo +## Architecture Diagram - +Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the CAPI operator for cluster management. + +![overview](/img/30000ft_view.png) + +## Prerequisites + +Before installing Rancher Turtles in your Rancher environment, you must disable Rancher's `embedded-cluster-api` functionality. This also includes cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. + +To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that removes the following: + +- Disables the `embedded-cluster-api` feature in Rancher. +- Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. ## Installing the Rancher Turtles Operator You can install the Rancher Turtles operator via the Rancher UI, or with Helm. The first method is recommended for most environments. :::caution + If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#via-helm-install). + ::: :::info prerequisite @@ -26,12 +39,10 @@ To simplify setting up Rancher for installing Rancher Turtles, the official Ranc - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. ::: -### Via Rancher UI +### Installing via the Rancher UI By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the CAPI Extension. -#### Installation - 1. Click **☰**. Under **Explore Cluster** in the left navigation menu, select **local**. 1. In the left navigation menu of the **Cluster Dashboard**, select **Apps > Repositories**. 1. Click **Create** to add a new repository. @@ -46,24 +57,25 @@ By adding the Turtles repository via the Rancher UI, Rancher can process the ins This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#via-helm-install). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). -The installation may take a few minutes and once completed you will see the following new deployments in the cluster: +The installation may take a few minutes and after completing you can see the following new deployments in the cluster: + - `rancher-turtles-system/rancher-turtles-controller-manager` - `rancher-turtles-system/rancher-turtles-cluster-api-operator` - `capi-system/capi-controller-manager` -### Via Helm Install +### Installing via Helm There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: -- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-capi-as-a-helm-dependency). +- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). - [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). -#### Install Rancher Turtles with `Cluster API (CAPI) Operator` as a Helm dependency +#### Installing Rancher Turtles with `Cluster API (CAPI) Operator` as a Helm dependency -1. The `rancher-turtles` chart is available in https://rancher.github.io/turtles. This Helm repository must be added before installation: +1. Add the Helm repository containing the `rancher-turtles` chart as the first step in installation: ```bash helm repo add turtles https://rancher.github.io/turtles @@ -73,26 +85,29 @@ helm repo update 2. As mentioned before, installing Rancher Turtles requires the [CAPI Operator](https://github.com/kubernetes-sigs/cluster-api-operator). The Helm chart can automatically install it with a minimal set of flags: ```bash -helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 \ +helm install rancher-turtles turtles/rancher-turtles --version \ -n rancher-turtles-system \ --dependency-update \ --create-namespace --wait \ --timeout 180s ``` -3. This operation could take a few minutes and once completed you can review the installed controllers listed below: +3. This operation could take a few minutes and after completing you can review the installed controllers listed below: + - `rancher-turtles-controller` - `capi-operator` :::note + - If `cert-manager` is already available in the cluster, disable its installation as a Rancher Turtles dependency. This prevents dependency conflicts: `--set cluster-api-operator.cert-manager.enabled=false` - For a list of Rancher Turtles versions, refer to the [Turtles release page](https://github.com/rancher/turtles/releases). + ::: This is the basic, recommended configuration, which manages the creation of a secret containing the required CAPI feature flags (`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL` enabled) in the core provider namespace. These feature flags are required to enable additional CAPI functionality. -If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including enabling the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL`. +If you need to override the default behavior and use an existing secret (or add custom environment variables), you can pass the secret name Helm flag. In this case, as a user, you are in charge of managing the secret creation and its content, including enabling the minimum required features: `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` and `EXP_MACHINE_POOL`. ```bash helm install ... @@ -117,16 +132,20 @@ stringData: ``` :::info + For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) + ::: -#### Install Rancher Turtles without `Cluster API (CAPI) Operator` as a Helm dependency +#### Installing Rancher Turtles without `Cluster API (CAPI) Operator` as a Helm dependency :::note + Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. + ::: -1. Add the `rancher-turtles` chart from the in https://rancher.github.io/turtles Helm repository before installation: +1. Add the Helm repository containing the `rancher-turtles` chart as the first step in installation: ```bash helm repo add turtles https://rancher.github.io/turtles @@ -136,7 +155,7 @@ helm repo update 2. Install the chart into the `rancher-turtles-system` namespace: ```bash -helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 +helm install rancher-turtles turtles/rancher-turtles --version -n rancher-turtles-system --set cluster-api-operator.enabled=false --set cluster-api-operator.cluster-api.enabled=false @@ -146,17 +165,21 @@ helm install rancher-turtles turtles/rancher-turtles --version v0.5.0 The previous commands tell Helm to ignore installing `cluster-api-operator` as a dependency. -3. This operation could take a few minutes and once completed you can review the installed controller listed below: +3. This operation could take a few minutes and after completing you can review the installed controller listed below: + - `rancher-turtles-controller` ## Uninstalling Rancher Turtles :::caution + When installing Rancher Turtles in your Rancher environment, by default, Rancher Turtles enables the CAPI Operator cleanup. This includes cleaning up CAPI Operator specific webhooks and deployments that otherwise cause issues with Rancher provisioning. -To simplify uninstalling Rancher Turtles (via Rancher Manager or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: +To simplify uninstalling Rancher Turtles (via Rancher or Helm command), the official Rancher Turtles Helm chart includes a `post-delete` hook that that removes the following: + - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks that are no longer needed. - Deletes the CAPI `deployments` that are no longer needed. + ::: To uninstall Rancher Turtles: @@ -168,12 +191,15 @@ helm uninstall -n rancher-turtles-system rancher-turtles --cascade foreground -- This may take a few minutes to complete. :::note + Remember that, if you use a different name for the installation or a different namespace, you may need to customize the command for your specific configuration. + ::: -Once Rancher Turtles is uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: +After Rancher Turtles is uninstalled, Rancher's `embedded-cluster-api` feature must be re-enabled: 1. Create a `feature.yaml` file, with `embedded-cluster-api` set to true: + ```yaml title="feature.yaml" apiVersion: management.cattle.io/v3 kind: Feature @@ -182,16 +208,16 @@ metadata: spec: value: true ``` + 2. Use `kubectl` to apply the `feature.yaml` file to the cluster: + ```bash kubectl apply -f feature.yaml ``` -## Architecture Diagram +## Demo -Below is a visual representation of the key components of Rancher Turtles and their relationship to Rancher Manager and the Rancher Cluster Agent. Understanding these components is essential for gaining insights into how Rancher leverages the CAPI operator for cluster management. - -![overview](/img/30000ft_view.png) + ## Security From f6ecdab0101cd20fa7ac6dab83e12c89a709b4de Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 8 Apr 2024 10:25:37 -0700 Subject: [PATCH 20/67] Added brief body text as a demo overview and moved the demo section into the installing via UI section as the demo primarily uses the Rancher UI to accomplish tasks. Signed-off-by: Sunil Singh --- .../cluster-api/overview.md | 18 ++++++------------ .../cluster-api/overview.md | 18 ++++++------------ .../cluster-api/overview.md | 18 ++++++------------ 3 files changed, 18 insertions(+), 36 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index c15753877cc..7abde17c1ba 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -31,14 +31,6 @@ If you already have the Cluster API (CAPI) Operator installed in your cluster, y ::: -:::info prerequisite -Before installing Rancher Turtles in your Rancher environment, you must disable Rancher's `embedded-cluster-api` functionality. This also includes cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. - -To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that removes the following: -- Disables the `embedded-cluster-api` feature in Rancher. -- Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. -::: - ### Installing via the Rancher UI By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the CAPI Extension. @@ -63,6 +55,12 @@ The installation may take a few minutes and after completing you can see the fol - `rancher-turtles-system/rancher-turtles-cluster-api-operator` - `capi-system/capi-controller-manager` +#### Demo + +This demo illustrates how to use the Rancher UI to install Rancher Turtles, create/import a CAPI cluster, and install monitoring on the cluster: + + + ### Installing via Helm There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: @@ -215,10 +213,6 @@ spec: kubectl apply -f feature.yaml ``` -## Demo - - - ## Security [SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index c15753877cc..7abde17c1ba 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -31,14 +31,6 @@ If you already have the Cluster API (CAPI) Operator installed in your cluster, y ::: -:::info prerequisite -Before installing Rancher Turtles in your Rancher environment, you must disable Rancher's `embedded-cluster-api` functionality. This also includes cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. - -To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that removes the following: -- Disables the `embedded-cluster-api` feature in Rancher. -- Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. -::: - ### Installing via the Rancher UI By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the CAPI Extension. @@ -63,6 +55,12 @@ The installation may take a few minutes and after completing you can see the fol - `rancher-turtles-system/rancher-turtles-cluster-api-operator` - `capi-system/capi-controller-manager` +#### Demo + +This demo illustrates how to use the Rancher UI to install Rancher Turtles, create/import a CAPI cluster, and install monitoring on the cluster: + + + ### Installing via Helm There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: @@ -215,10 +213,6 @@ spec: kubectl apply -f feature.yaml ``` -## Demo - - - ## Security [SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index c15753877cc..7abde17c1ba 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -31,14 +31,6 @@ If you already have the Cluster API (CAPI) Operator installed in your cluster, y ::: -:::info prerequisite -Before installing Rancher Turtles in your Rancher environment, you must disable Rancher's `embedded-cluster-api` functionality. This also includes cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. - -To simplify setting up Rancher for installing Rancher Turtles, the official Rancher Turtles Helm chart includes a `pre-install` hook that removes the following: -- Disables the `embedded-cluster-api` feature in Rancher. -- Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. -::: - ### Installing via the Rancher UI By adding the Turtles repository via the Rancher UI, Rancher can process the installation and configuration of the CAPI Extension. @@ -63,6 +55,12 @@ The installation may take a few minutes and after completing you can see the fol - `rancher-turtles-system/rancher-turtles-cluster-api-operator` - `capi-system/capi-controller-manager` +#### Demo + +This demo illustrates how to use the Rancher UI to install Rancher Turtles, create/import a CAPI cluster, and install monitoring on the cluster: + + + ### Installing via Helm There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: @@ -215,10 +213,6 @@ spec: kubectl apply -f feature.yaml ``` -## Demo - - - ## Security [SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. From d156476d16d10b7c6e7b81c3cf1ff0bf74e6bf73 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 8 Apr 2024 10:32:19 -0700 Subject: [PATCH 21/67] Fixing markdown links and syncing across versions. Signed-off-by: Sunil Singh --- docs/integrations-in-rancher/cluster-api/overview.md | 11 ++++++----- .../integrations-in-rancher/cluster-api/overview.md | 11 ++++++----- .../integrations-in-rancher/cluster-api/overview.md | 11 ++++++----- 3 files changed, 18 insertions(+), 15 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 7abde17c1ba..66887cc8602 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -27,7 +27,7 @@ You can install the Rancher Turtles operator via the Rancher UI, or with Helm. T :::caution -If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#via-helm-install). +If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#installing-via-helm). ::: @@ -47,7 +47,7 @@ By adding the Turtles repository via the Rancher UI, Rancher can process the ins 1. Click **Rancher Turtles - the Cluster API Extension**. 1. Click **Install > Next > Install**. -This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#via-helm-install). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). +This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). The installation may take a few minutes and after completing you can see the following new deployments in the cluster: @@ -64,10 +64,11 @@ This demo illustrates how to use the Rancher UI to install Rancher Turtles, crea ### Installing via Helm There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: -- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). -- [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). -The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. +- [Install Rancher Turtles with CAPI Operator as a dependency](#installing-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). +- [Install Rancher Turtles without CAPI Operator](#installing-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). + +The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#installing-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index 7abde17c1ba..66887cc8602 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -27,7 +27,7 @@ You can install the Rancher Turtles operator via the Rancher UI, or with Helm. T :::caution -If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#via-helm-install). +If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#installing-via-helm). ::: @@ -47,7 +47,7 @@ By adding the Turtles repository via the Rancher UI, Rancher can process the ins 1. Click **Rancher Turtles - the Cluster API Extension**. 1. Click **Install > Next > Install**. -This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#via-helm-install). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). +This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). The installation may take a few minutes and after completing you can see the following new deployments in the cluster: @@ -64,10 +64,11 @@ This demo illustrates how to use the Rancher UI to install Rancher Turtles, crea ### Installing via Helm There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: -- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). -- [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). -The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. +- [Install Rancher Turtles with CAPI Operator as a dependency](#installing-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). +- [Install Rancher Turtles without CAPI Operator](#installing-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). + +The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#installing-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index 7abde17c1ba..66887cc8602 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -27,7 +27,7 @@ You can install the Rancher Turtles operator via the Rancher UI, or with Helm. T :::caution -If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#via-helm-install). +If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#installing-via-helm). ::: @@ -47,7 +47,7 @@ By adding the Turtles repository via the Rancher UI, Rancher can process the ins 1. Click **Rancher Turtles - the Cluster API Extension**. 1. Click **Install > Next > Install**. -This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#via-helm-install). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). +This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). The installation may take a few minutes and after completing you can see the following new deployments in the cluster: @@ -64,10 +64,11 @@ This demo illustrates how to use the Rancher UI to install Rancher Turtles, crea ### Installing via Helm There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: -- [Install Rancher Turtles with CAPI Operator as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). -- [Install Rancher Turtles without CAPI Operator](#install-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). -The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#install-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. +- [Install Rancher Turtles with CAPI Operator as a dependency](#installing-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). +- [Install Rancher Turtles without CAPI Operator](#installing-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). + +The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#installing-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). From 51b462f97d88ba6e2c8f5132032741c8edd64219 Mon Sep 17 00:00:00 2001 From: siye <503630985@qq.com> Date: Tue, 9 Apr 2024 01:46:52 +0800 Subject: [PATCH 22/67] =?UTF-8?q?Update=20missing=20Chinese=20translation?= =?UTF-8?q?=20about=20vsphere/workload/project/quo=E2=80=A6=20(#1211)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Update missing Chinese translation about current vsphere/workload/project/quota/loadbalancer/helm/backup/app/autoscaler * Update missing Chinese translation about version-2.8 vsphere/workload/project/quota/loadbalancer/helm/backup/app/autoscaler --- .../advanced-user-guides.md | 11 + .../manage-project-resource-quotas.md | 50 ++++ .../manage-projects/manage-projects.md | 41 ++++ .../backup-restore-and-disaster-recovery.md | 100 ++++++++ .../deploy-apps-across-clusters.md | 21 ++ .../helm-charts-in-rancher.md | 216 ++++++++++++++++++ .../horizontal-pod-autoscaler.md | 30 +++ .../kubernetes-resources-setup.md | 67 ++++++ .../load-balancer-and-ingress-controller.md | 65 ++++++ .../workloads-and-pods/workloads-and-pods.md | 82 +++++++ .../nutanix/nutanix.md | 21 ++ .../vsphere/vsphere.md | 60 +++++ .../advanced-user-guides.md | 11 + .../manage-project-resource-quotas.md | 50 ++++ .../manage-projects/manage-projects.md | 41 ++++ .../backup-restore-and-disaster-recovery.md | 100 ++++++++ .../deploy-apps-across-clusters.md | 21 ++ .../helm-charts-in-rancher.md | 216 ++++++++++++++++++ .../horizontal-pod-autoscaler.md | 30 +++ .../kubernetes-resources-setup.md | 67 ++++++ .../load-balancer-and-ingress-controller.md | 65 ++++++ .../workloads-and-pods/workloads-and-pods.md | 82 +++++++ .../nutanix/nutanix.md | 21 ++ .../vsphere/vsphere.md | 60 +++++ 24 files changed, 1528 insertions(+) create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/advanced-user-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/advanced-user-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/advanced-user-guides.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/advanced-user-guides.md new file mode 100644 index 00000000000..ef7e395ad3c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/advanced-user-guides.md @@ -0,0 +1,11 @@ +--- +title: 高级用户指南 +--- + + + + + +高级用户指南是“问题导向”的文档，用户可以从中学习如何解决问题。高级用户指南与新用户指南的主要区别在于，高级用户指南面向更有经验或更高级的用户，这些用户对文档有更多的技术需求，而且已经了解 Rancher 及其功能。他们知道自己需要做什么，只是需要额外的指导来完成更复杂的任务。 + +应该注意的是，新用户指南和高级用户指南都没有提供详细的解释或讨论（这些文档不包括在本部分）。操作指南侧重于引导用户通过可重复、有效的步骤来学习新技能、掌握某些操作或解决某些问题。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md new file mode 100644 index 00000000000..ce57b2db349 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md @@ -0,0 +1,50 @@ +--- +title: 项目资源配额 +--- + + + + + +如果多个团队共享一个集群，某个团队可能会使用过多的可用资源，例如 CPU、内存、存储、服务、Kubernetes 对象（如 Pod 或 Secret）等。你可以应用 _资源配额_ 来防止过度消耗资源。资源配额是 Rancher 用来限制项目或命名空间可用资源的功能。 + +本文介绍如何在现有项目中创建资源配额。 + +你也可以在创建新项目时设置资源配额。有关详细信息，请参阅[创建新项目](../../../new-user-guides/manage-clusters/projects-and-namespaces.md#创建项目)。 + +Rancher 中的资源配额包含与 [Kubernetes 原生版本](https://kubernetes.io/docs/concepts/policy/resource-quotas/)相同的功能。Rancher 还扩展了资源配额的功能，从而让你将资源配额应用于项目。有关资源配额如何与 Rancher 中的项目一起使用的详细信息，请参阅[此页面](about-project-resource-quotas.md)。 + +### 将资源配额应用于现有项目 + +修改资源配额的使用场景如下： + +- 限制某个项目和项目下的命名空间能使用的资源 +- 在资源配额已生效的情况下，对项目可用的资源进行扩容或缩容 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面，进入要应用资源配额的集群，然后单击 **Explore**。 +1. 单击**集群 > 项目/命名空间**。 +1. 确保 **Projects/Namespaces** 页面处于 **Group by Project** 视图模式。 + ![Screenshot highlighting the "Group by Project" icon, above the list of projects. It resembles a folder.](/img/edit-project-config-for-resource-quotas-group-by-project.png) + +1. 找到要添加资源配额的项目，选择与项目名称同行的 **⋮**。 + ![Screenshot highlighting triple dots icon at the end of the same row as the project name.](/img/edit-project-config-for-resource-quotas-dots.png) + +1. 选择**编辑配置**。 + +1. 展开**资源限额**并单击**添加资源**。你也可以编辑现有配额。 + +1. 选择资源类型。有关类型的更多信息，请参阅[配额类型参考](resource-quota-types.md)。 + +1. 输入**项目限制**和**命名空间默认限制**的值。 + + | 字段 | 描述 | + | ----------------------- | -------------------------------------------------------------------------------------------------------- | + | 项目限制 | 项目的总资源限制。 | + | 命名空间默认限制 | 每个命名空间的默认资源限制。此限制会沿用到项目中的每个命名空间。项目中所有命名空间的限制之和不应超过项目限制。 | + +1. **可选**：添加更多配额。 + +1. 单击**创建**。 + +**结果**：资源配额已应用到你的项目和命名空间。如果你后续需要添加更多命名空间，Rancher 会验证项目是否可以容纳该命名空间。如果项目无法分配资源，你仍然可以创建命名空间，但命名空间将获得的资源配额为 0。然后 Rancher 将不允许你创建任何受此配额限制的资源。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md new file mode 100644 index 00000000000..1ce6b33f141 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md @@ -0,0 +1,41 @@ +--- +title: 项目管理 +--- + + + + + +_项目_ 是 Rancher 中引入的对象，可帮助你更有组织地管理 Kubernetes 集群中的命名空间。你可以使用项目创建多租户集群，这种集群允许一组用户共享相同的底层资源来创建应用，而应用之间不会相互影响。 + +在层次结构方面： + +- 集群包含项目 +- 项目包含命名空间 + +在 Rancher 中，你可以使用项目将多个命名空间作为一个实体进行管理。在原生 Kubernetes（没有项目这个概念）中，RBAC 或集群资源等功能被分配给了各个命名空间。如果集群中的多个命名空间需要分配同样的访问权限，分配权限会变得非常繁琐。即使所有命名空间都需要相同的权限，但也无法使用一个操作中将这些权限应用于所有命名空间。你必须重复地将这些权限分配给每个命名空间。 + +而 Rancher 通过引入项目的概念，通过允许你在项目级别应用资源和访问权限。然后，项目中的每个命名空间都会继承这些资源和策略。因此你只需将资源和策略分配给项目即可，不需要将它们分配给每个单独的命名空间。 + +你可以使用项目执行以下操作： + +- [为用户分配一组命名空间的访问权限](../../new-user-guides/add-users-to-projects.md) +- 为用户分配[项目中的特定角色](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色)。角色可以是所有者、成员、只读或[自定义](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/custom-roles.md) +- [设置资源配额](manage-project-resource-quotas/manage-project-resource-quotas.md) +- [管理命名空间](../../new-user-guides/manage-namespaces.md) +- [配置工具](../../../reference-guides/rancher-project-tools.md) +- [配置 Pod 安全策略](manage-pod-security-policies.md) + +### 授权 + +非管理者用户只有在[管理员](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md)、[集群所有者或成员](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#集群角色)或[项目所有者](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色)将非管理员用户添加到项目的**成员**选项卡后，才能获取项目的访问权限。 + +创建项目的人自动成为[项目所有者](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色)。 + +## 在项目之间切换 + +要在项目之间切换，请使用导航栏中的下拉菜单。你也可以直接在导航栏中切换项目： + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面，进入要切换项目的集群然后点击 **Explore**。 +1. 在顶部导航栏中，选择要打开的项目。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md new file mode 100644 index 00000000000..d99c8f91f92 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md @@ -0,0 +1,100 @@ +--- +title: 备份和灾难恢复 +keywords: [rancher 备份还原, rancher 备份与恢复, 备份恢复 rancher, rancher 备份与恢复 rancher] +--- + + + + + +在本节中，你将学习如何创建 Rancher 的备份，如何从备份中恢复 Rancher，以及如何将 Rancher 迁移到新的 Kubernetes 集群。 + +`rancher-backup` operator 可以用来备份和恢复任何 Kubernetes 集群上的 Rancher。这个应用是一个 Helm Chart，可以通过 Rancher 的 **Apps** 页面或使用 Helm CLI 部署。你可以访问[本页面](https://github.com/rancher/charts/tree/release-v2.6/charts/rancher-backup)获取 `rancher-backup` Helm Chart。 + +backup-restore operator 需要安装在 local 集群上，并且只对 Rancher 应用进行备份。备份和恢复操作仅在本地 Kubernetes 集群中执行。 + + +## 备份和恢复 Docker 安装的 Rancher + +对于使用 Docker 安装的 Rancher，请参见[备份](back-up-docker-installed-rancher.md)和[恢复](restore-docker-installed-rancher.md)对 Rancher 进行备份和恢复。 + +## 备份和恢复原理 + +`rancher-backup` operator 引入了三个自定义资源，分别是 Backups、Restores 和 ResourceSets。将以下集群范围的自定义资源定义添加到集群中： + +- `backups.resources.cattle.io` +- `resourcesets.resources.cattle.io` +- `restores.resources.cattle.io` + +ResourceSet 定义了需要备份哪些 Kubernetes 资源。由于备份 Rancher 所需的值是预设的，因此 ResourceSet 无法通过 Rancher UI 进行配置。请不要修改此 ResourceSet。 + +在创建 Backup 自定义资源时，`rancher-backup` operator 调用 `kube-apiserver` 来获取 Backup 自定义资源引用的 ResourceSet（即预设的 `rancher-resource-set`）资源。 + +然后，operator 以 .tar.gz 格式创建备份文件，并将其存储在 Backup 资源中配置的位置。 + +在创建 Restore 自定义资源时，operator 访问 Restore 指定的 tar.gz 备份文件，并从该文件恢复应用。 + +你可以使用 Rancher UI 或 `kubectl apply` 来创建 Backup 和 Restore 自定义资源。 + +:::note + +请参见[此处](migrate-rancher-to-new-cluster.md#2-使用-restore-自定义资源来还原备份)获取在 Rancher 2.6.3 中将现有备份文件恢复到 v1.22 集群的帮助。 + +::: + +## 安装 rancher-backup operator + +你可以使用 Rancher UI 或 Helm CLI 来安装 `rancher-backup` operator。两种安装方法都将 `rancher-backup` Helm Chart 安装在运行 Rancher Server 的 Kubernetes 集群上。它是集群管理员独有的功能，仅适用于 **local** 集群。（*如果你在 Rancher UI 中没有看到 `rancher-backup`，你可能选择了错误的集群。*） + +:::note + +使用 backup-restore-operator 执行恢复后，Fleet 中会出现一个已知问题：用于 clientSecretName 和 helmSecretName 的密文不包含在 Fleet 的 Git 仓库中。请参见[此处](../../../integrations-in-rancher/fleet/overview.md#故障排除)获得解决方法。 + +::: + +### 使用 Rancher UI 安装 rancher-backup + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面上，转到 `local` 集群并单击 **Explore**。 +1. 在左侧导航栏中，单击 **Apps > Charts**。 +1. 点击 **Rancher 备份**。 +1. 单击**安装**。 +1. 可选：配置默认存储位置。如需获取帮助，请参见[配置](../../../reference-guides/backup-restore-configuration/storage-configuration.md)。 +1. 单击**安装**。 + +**结果**：`rancher-backup` operator 已安装。 + +在**集群仪表板**中，你可以看到列在 **Deployments** 下的 `rancher-backup` operator。 + +如果需要在 Rancher 中配置备份应用，在左侧导航栏中单击 **Rancher 备份**。 + +### RBAC + +只有 Rancher 管理员和本地集群的所有者可以： + +* 安装 Chart +* 看到 Backup 和 Restore CRD 的导航链接 +* 通过分别创建 Backup CR 和 Restore CR 执行备份和恢复 +* 列出目前已执行的备份和恢复操作 + +## 备份 Rancher + +备份是通过创建 Backup 自定义资源实现的。如需查看教程，请参见[本页面](back-up-rancher.md)。 + +## 还原 Rancher + +还原是通过创建 Restore 自定义资源实现的。如需查看教程，请参见[本页面](restore-rancher.md)。 + +## 将 Rancher 迁移到新集群 + +你可以按照[这些步骤](migrate-rancher-to-new-cluster.md)执行迁移。 + +## 默认存储位置配置 + +配置一个用户保存备份文件的默认存储位置。有多种选项可以配置，例如将与 S3 兼容的对象存储作为单个备份文件的存储后端，或在安装 `backup-restore-operator` Helm Chart 时选择一个现有的 `StorageClass`。你还可以选择在每次备份时覆盖配置的存储位置，但这仅限于使用与 S3 兼容的对象存储或 Minio 对象存储。 + +如需了解各个选项的配置，请参见[本页面](../../../reference-guides/backup-restore-configuration/storage-configuration.md)。 + +### 示例 YAML 文件：Rancher Backup Helm Chart + +当使用 Helm CLI 安装时，可以使用示例 [values.yaml 文件](../../../reference-guides/backup-restore-configuration/storage-configuration.md#rancher-backup-helm-chart-的示例-valuesyaml) 来配置 `rancher-backup-operator`。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md new file mode 100644 index 00000000000..4d4d465d136 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md @@ -0,0 +1,21 @@ +--- +title: 跨集群部署应用 +--- + + + + + +不同版本的 Rancher 提供了几种不同的方式来部署跨集群应用。 + +## Fleet + +Rancher v2.5 及更高版本使用 Fleet 跨集群部署应用 + +使用 Fleet 的持续交付是大规模的 GitOps。如需更多信息，请参阅 [Fleet](fleet.md)。 + +### 多集群应用 + +在 v2.5 之前的 Rancher 中，多集群应用功能用于跨集群部署应用。多集群应用功能已弃用，但仍可作为旧版功能使用。 + +详情请参阅[此文档](multi-cluster-apps.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md new file mode 100644 index 00000000000..48ecc783bee --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md @@ -0,0 +1,216 @@ +--- +title: Helm Charts 和 Apps +--- + + + + + +在本节中，你将学习如何在 Rancher 中管理 Helm Chart 仓库和应用。 + +## Helm Charts 在 Rancher 中的工作原理 + +在 Rancher 中 Helm chart 仓库是使用 **Apps** 进行管理的。 + +Rancher 使用应用商店系统导入一系列的 charts 包到仓库里，然后使用这些 charts 来部署自定义的 Kubernetes 应用程序或 Rancher 的工具，如监控（Monitoring）或 Istio。Rancher 工具预先集成进了仓库，并通过独立的 Helm chart 进行部署。如果你有自己的额外仓库只需要添加到当前的集群中就可以正常部署。 + +### Catalogs, Apps, Rancher UI + +[在 Rancher v2.4 及更早版本中](/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md), 存储部署的应用程序的仓库被称为 "catalogs"。这些仓库是通过 UI 的 **Catalogs** 进行管理的。 + +Rancher v2.5 用新的 **Apps & Marketplace** 功能替代了之前的 **应用商店**。 + +从 Rancher v2.6.5 开始, **Apps & Marketplace** 功能在 UI 中被命名为 **Apps**。 + +### Versioning Scheme + +Rancher chart 的版本设计方案主要围绕 charts 的主要版本以及与上游 charts 相关的 `+up` 注释进行设计。 + +**主要版本:** charts 的主要版本与 Rancher 的特定次要版本相关联。当你升级到新的 Rancher 次要版本时，你应该确保所有的功能 charts 也被升级对应的主版本。 + +**基于上游的 Charts:** 在升级时，确保上游 chart 的版本与你的 Rancher 版本兼容。chart 的 `+up` 注释指示 Rancher chart 正在跟踪的上游版本。例如，`100.x.x+up16.6.0` 的监控（Monitoring）跟踪上游的 kube-prometheus-stack `16.6.0` 并附带一些额外的 Rancher 补丁。 + +在升级 Rancher 版本时，不要降级你正在使用的 chart 版本。例如，如果你在 Rancher v2.5 中使用的监控版本比 `16.6.0` 还要高，你不应该升级到 `100.x.x+up16.6.0`，而是应该在下一个发布中升级到适当的版本。 + +#### 预发布版本 + +预发布版本遵循[语义化版本 2.0.0 ](https://semver.org/)的[规范](https://semver.org/#spec-item-9)。 例如，版本为 `0.1.3-dev.12ab4f` 的 Helm chart 被认为是一个预发布版本。预发布版本默认不显示，必须进行配置以显示。 + +如何显示预发布版本： + +1. 点击右上角的用户头像。 +2. 点击 **Preferences**. +3. 在 **Helm Charts** 下, 选择 **Include Prerelease Versions**. + +### 版本控制方案 + +| **Name** | **支持的最低版本** | **支持的最高版本** | +| ---------------- | ------------ | ------------ | +| external-ip-webhook | 100.0.0+up1.0.0 | 100.0.1+up1.0.1 | +| harvester-cloud-provider | 100.0.2+up0.1.12 | 100.0.2+up0.1.12 | +| harvester-csi-driver | 100.0.2+up0.1.11 | 100.0.2+up0.1.11 | +| neuvector | 100.0.0+up2.2.0 | 100.0.0+up2.2.0 | +| rancher-alerting-drivers | 100.0.0 | 100.0.2 | +| rancher-backup | 2.0.1 | 2.1.2 | +| rancher-cis-benchmark | 2.0.1 | 2.0.4 | +| rancher-gatekeeper | 100.0.0+up3.6.0 | 100.1.0+up3.7.1 | +| rancher-istio | 100.0.0+up1.10.4 | 100.3.0+up1.13.3 | +| rancher-logging | 100.0.0+up3.12.0 | 100.1.2+up3.17.4 | +| rancher-longhorn | 100.0.0+up1.1.2 | 100.1.2+up1.2.4 | +| rancher-monitoring | 100.0.0+up16.6.0 | 100.1.2+up19.0.3 | +| rancher-sriov (experimental) | 100.0.0+up0.1.0 | 100.0.3+up0.1.0 | +| rancher-vsphere-cpi | 100.3.0+up1.2.1 | 100.3.0+up1.2.1 | +| rancher-vsphere-csi | 100.3.0+up2.5.1-rancher1 | 100.3.0+up2.5.1-rancher1 | +| rancher-wins-upgrader | 0.0.100 | 100.0.1+up0.0.1 | + +## 访问 Charts + +**Charts** 页面包含所有 Rancher、Rancher 合作伙伴和自定义 Chart。你可以通过选择左侧的下拉菜单来筛选 Chart： + +* Rancher 工具（例如 Logging 或 Monitoring）包含在 **Rancher** 标签下 +* Partner Chart 位于 **Partner** 标签下 +* 自定义 Chart 将显示在仓库的名称下 + +所有这三种类型都以相同的方式部署和管理。 + +:::note +由 Cluster Manager （旧版 Rancher UI 中的全局视图）管理的应用应继续仅由 Cluster Manager 管理，而在新 UI 中使用 Apps 管理的应用则仅能由 Apps 管理。 +::: + +访问 **Charts** 页面: + +1. 点击 **☰ > Cluster Management**。 +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中, 点击 **Apps > Charts**。 + +### 管理仓库 + +**Repositories** 页面列出了你的 Helm 仓库。 这包括传统的具有 index.yaml 的 Helm 端点，以及克隆并指向特定分支的 Git 仓库。要使用自定义 Charts，在这里可以添加你的仓库。添加仓库后，你可以在 **Charts** 页面中访问自定义 Charts，这些 Charts 将列在仓库的名称下。 + +访问 **Repositories** 页面: + +1. 点击 **☰ > Cluster Management**. +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 + +### 添加自定义 Git 仓库 + +要添加一个包含你的 Helm Charts 或集群模板定义的自定义 Git 仓库： + +1. 点击 **☰ > Cluster Management**。 +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 +4. 点击 **Create**。 +5. 选择 **Git repository containing Helm chart...**。 +6. 你必须输入名称和 Git 仓库的 URL。其他配置项包括描述，都是可选的。如果你不想设置默认的分支，可以输入你想要使用的分支名称。通常，默认分支名为 `main` 或 `master`。 +7. 点击 **Create** 添加。 + +在 Rancher 中添加 Charts 仓库后，它将立即生效。 + +### 添加自定义 Helm Chart 仓库 + +你可以将自己的 Helm chart 仓库添加到 Rancher。为了能正确添加 http 的 Helm Chart 仓库，你需要提供 chart 的服务器并能够响应 GET 请求并提供 YAML 文件和 tar 包。 + +有关 Helm chart 仓库的更多信息，请参阅 [官方 Helm 文档](https://helm.sh/docs/topics/chart_repository/)。 + +要将自定义 Helm chart 仓库添加到 Rancher： + +1. 点击 **☰ > Cluster Management**。 +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 +4. 点击 **Create**。 +5. 选择 **http(s) URL to an index generated by Helm**. +6. 输入仓库名称和 chart 的 index URL 地址。 +7. 点击 **Create** 添加。 + +### 添加私有 Git/Helm Chart 仓库 + +你可以使用 SSH 密钥凭据或 HTTP 基础认证秘密（如用户名和密码）添加私有 Git 或 Helm chart 仓库。 + +### 向仓库添加私有 CA + +向 Helm chart 仓库添加私有 CA，你必须将 DER 格式的 CA 证书的 base64 编码副本添加到 Chart 仓库的 `spec.caBundle 字段`，例如 `openssl x509 -outform der -in ca.pem | base64 -w0`。无论是基于 Git 还是 HTTP 的仓库，操作步骤都是相同的 + +1. 点击 **☰**。在左侧导航菜单的 **Explore Cluster**, 选择一个集群。 +2. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 +3. 找到你想要向其添加私有 CA 证书的 Git 或 HTTP 的仓库。点击 **⋮ > Edit YAML**。 +4. 设置 `caBundle` 值，如以下示例： + + ```yaml + [...] + spec: + caBundle: + MIIFXzCCA0egAwIBAgIUWNy8WrvSkgNzV0zdWRP79j9cVcEwDQYJKoZIhvcNAQELBQAwPzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRQwEgYDVQQKDAtNeU9yZywgSW5jLjENMAsGA1UEAwwEcm9vdDAeFw0yMTEyMTQwODMyMTdaFw0yNDEwMDMwODMyMT + ... + nDxZ/tNXt/WPJr/PgEB3hQdInDWYMg7vGO0Oz00G5kWg0sJ0ZTSoA10ZwdjIdGEeKlj1NlPyAqpQ+uDnmx6DW+zqfYtLnc/g6GuLLVPamraqN+gyU8CHwAWPNjZonFN9Vpg0PIk1I2zuOc4EHifoTAXSpnjfzfyAxCaZsnTptimlPFJJqAMj+FfDArGmr4= + [...] + ``` + +:::note 带有认证的 Helm chart 仓库 + +Repo.Spec 包含一个 `disableSameOriginCheck` 值，该值允许用户绕过相同源的检查，将仓库身份认证信息作为基本 Auth 标头与所有 API 调用一起发送。不建议采用这种做法，但这可以用作非标准 Helm Chart 仓库（例如重定向到不同源 URL 的仓库）的临时解决方案。 + +要将此功能用于现有 Helm Chart 仓库，请按照前面的步骤编辑 YAML。在 YAML 文件的 `spec` 部分，添加 `disableSameOriginCheck` 并将其设置为 `true`： + +```yaml +[...] +spec: + disableSameOriginCheck: true +[...] +``` + +::: + +### Helm 兼容性 + +仅支持 Helm 3 兼容 Chart 。 + +### 部署和升级 Chart + +安装和部署 chart: + +1. 点击 **☰ > Cluster Management**。 +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Charts**。 +4. 选择一个 chart 点击 **Install**。 + +Rancher 和 Partner Chart 可能通过自定义页面或 questions.yaml 文件进行额外的配置，但所有 Chart 安装都可以修改 values.yaml 和其他基本设置。单击安装后，将部署一个 Helm 操作作业，并显示该作业的控制台。 + +要查看所有最近的更改，点击左侧导航栏菜单中的 **APPs > Recent Operations**。你可以查看已进行的调用、条件、事件和日志 + +安装 Chart 后，点击左侧导航栏菜单中的 **Apps > Installed Apps**。在本节中，你可以升级或删除安装，并查看更多详细信息。选择升级时，呈现的形式和数值与安装相同。 + +大多数 Rancher 工具在 **Apps** 下方的工具栏中都有额外的页面，以帮助你管理和使用这些功能。这些页面包括指向仪表板的链接、可轻松添加自定义资源的表单以及其他信息。 + +:::caution + +如果在升级前使**自定义 Helm 选项**，并且你的 Chart 中包含不可更改的字段，使用 `--force` 选项可能会导致错误。这是因为 Kubernetes 中的某些对象一旦创建就无法更改。要避免该错误，你可以： + +* 使用默认升级选项（即不要使用 `--force` 选项） +* 卸载现有 Chart 并安装升级后的 Chart +* 在执行强制升级之前删除集群中具有不可更改字段的资源 + +::: + +#### 旧版应用 + +在 **Apps > Installed Apps** 页面上，旧版应用没有升级按钮。 + +如果你想升级已安装的旧版应用， 必须启用[旧版功能](../../advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 如果你在升级 Rancher 之前已经运行了旧版应用，此标志会自动启用。 + +1. 如果尚未启用，启用[旧版功能](../../advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 +2. 点击 **☰ > Cluster Management**。 +3. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +4. 点击 **Legacy > Project > Apps**. + +如果在 **Legacy > Project** 下看不到 **Apps** ,请点击顶部导航栏的 **project/namespace** 搜索栏，并从下拉菜单中选择相关项目。 + +要升级旧版多集群应用： + +1. 点击 **☰**. +2. 在 **Legacy Apps** 下点击 **Multi-cluster Apps**. + + +### 限制 + +Rancher CLI 不能用于安装仪表板应用程序或 Rancher 功能 Chart。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md new file mode 100644 index 00000000000..b50aecb39a9 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md @@ -0,0 +1,30 @@ +--- +title: Horizontal Pod Autoscaler +description: 了解 Pod 水平自动扩缩 (HPA)。如何管理 HPA 以及如何使用服务部署来进行测试 +--- + + + + + +[Horizontal Pod Autoscaler（HPA）](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)是一项 Kubernetes 功能，用于将集群配置为自动扩缩其运行的服务。 + +Rancher 提供了一些附加功能来帮助管理 HPA，具体取决于 Rancher 的版本。 + +你可以使用 Rancher UI 创建、管理和删除 HPA。仅在 `autoscaling/v2beta2` API 中支持 HPA。 + +## 管理 HPA + +管理 HPA 的方式因你的 Kubernetes API 版本而异： + +- **Kubernetes API 版本 autoscaling/V2beta1**：允许根据应用程序的 CPU 和内存利用率自动扩缩 pod。 +- **Kubernetes API 版本 autoscaling/V2beta2**：允许根据 CPU 和内存利用率以及自定义指标自动扩缩 pod。 + +你可以使用 Rancher UI 创建、管理和删除 HPA。在 Rancher UI 中，你可以将 HPA 配置为根据 CPU 和内存利用率进行扩缩。有关详细信息，请参阅[使用 Rancher UI 管理 HPA](manage-hpas-with-ui.md)。如需根据自定义指标进行 HPA，你仍然需要使用 `kubectl`。有关详细信息，请参阅[配置 HPA 以使用 Prometheus 自定义指标进行扩缩](manage-hpas-with-kubectl.md#使用资源指标cpu-和内存配置-hpa-以进行扩缩)。 + +在 Rancher 2.0.7 及更高版本中创建的集群自动满足使用 HPA 的所有要求（metrics-server 和 Kubernetes 集群配置）。 +## 使用服务部署测试 HPA + +你可以转到你的项目并单击 **资源 > HPA** 来查看​​ HPA 当前的副本数。有关详细信息，请参阅[获取 HPA 指标和状态](manage-hpas-with-ui.md)。 + +你还可以使用 `kubectl` 来获取你使用负载测试工具测试的 HPA 的状态。有关详细信息，请参阅[使用 kubectl 测试 HPA](test-hpas-with-kubectl.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md new file mode 100644 index 00000000000..4dd86bb2a6c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md @@ -0,0 +1,67 @@ +--- +title: Kubernetes 资源 +--- + + + + + +你可以在 Rancher UI 中查看和操作 Kubernetes 集群中的所有自定义资源和 CRD。 + +## 工作负载 + +使用[工作负载](workloads-and-pods/workloads-and-pods.md)将应用部署到集群节点，工作负载是包含用于运行应用的 pod 的对象，以及为部署行为设置规则的元数据。工作负载可以部署在集群范围内，也可以部署在一个命名空间内。 + +部署工作负载时，你可以使用任何镜像进行部署。可供选择的[工作负载类型](workloads-and-pods/workloads-and-pods.md#工作负载类型)有多种，工作负载类型决定了你的应用程序的运行方式。 + +在工作负载部署之后，你可以继续使用它。你可以： + +- 将工作负载[升级](workloads-and-pods/upgrade-workloads.md)到它运行的应用的更新版本。 +- 如果升级出现问题，将工作负载[回滚](workloads-and-pods/roll-back-workloads.md)到以前的版本。 +- [添加一个 sidecar](workloads-and-pods/add-a-sidecar.md)，这是一个支持主要工作负载的工作负载。 + +## 负载均衡和 Ingress + +### 负载均衡器 + +启动应用程序后，它仅在集群中可用。无法从外部访问它。 + +如果你希望你的应用程序可以从外部访问，则必须向集群添加负载均衡器。如果用户知道负载均衡器的 IP 地址和应用的端口号，负载均衡器可以为外部连接创建一个访问集群的网关。 + +Rancher 支持两种类型的负载均衡器： + +- [Layer-4 负载均衡器](load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing.md#layer-4-load-balancer#四层负载均衡器) +- [Layer-7 负载均衡器](load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing.md#七层负载均衡器) + +有关详细信息，请参阅[负载均衡器](load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing.md)。 + +#### Ingress + +负载均衡器只能处理每个 service 的一个 IP 地址。换言之，如果你在集群中运行了多个 service，则必须为每个 service 配备一个负载均衡器。运行多个负载均衡器的花费可能非常高昂。因此，你可以使用 Ingress 来解决此问题。 + +Ingress 是一组充当负载均衡器的规则。Ingress 与一个或多个 Ingress Controller 一起动态路由 service 的请求。Ingress 收到请求时，集群中的 Ingress Controller 会对负载均衡器进行配置，从而根据你配置的 service 子域或路径规则将请求定向到正确的 service。 + +有关详细信息，请参阅 [Ingress](load-balancer-and-ingress-controller/add-ingresses.md)。 + +在项目中使用 Ingress 时，你可以通过设置全局 DNS 条目来将 Ingress 主机名编程到外部 DNS。 + +## 服务发现 + +使用负载均衡器和/或 Ingress 将集群公开给外部请求后，你只能通过 IP 地址访问集群。要创建可解析的主机名，你必须创建服务记录，该记录将 IP 地址、外部主机名、DNS 记录别名、工作负载或标记的 pod 映射到特定主机名。 + +有关详细信息，请参阅[服务发现](create-services.md)。 + +## 应用程序 + +除了启动应用程序的各个组件外，你还可以使用 Rancher 应用商店来启动应用，即 Helm Chart。 + +## Kubernetes 资源 + +在 Rancher 项目或命名空间的上下文中，_资源_ 是支持 Pod 操作的文件和数据。在 Rancher 中，证书、镜像仓库和密文都被视为资源。但是，Kubernetes 将资源划分为不同类型的[密文（secret）](https://kubernetes.io/docs/concepts/configuration/secret/)。因此，在单个项目或命名空间中，各个资源必须具有唯一的名称以避免冲突。资源主要用于承载敏感信息，但也有其他用途。 + +资源包括： + +- [证书](encrypt-http-communication.md)：用于加密/解密进入或离开集群的数据的文件。 +- [ConfigMap](configmaps.md)：存储一般配置信息的文件，例如一组配置文件。 +- [密文](secrets.md)：存储密码、token 或密钥等敏感数据的文件。 +- [镜像仓库](kubernetes-and-docker-registries.md)：携带用于验证私有镜像仓库的凭证的文件。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md new file mode 100644 index 00000000000..00ae5a04894 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md @@ -0,0 +1,65 @@ +--- +title: 在 Rancher 中设置负载均衡器和 Ingress Controller +description: 了解如何设置负载均衡器和 Ingress Controller 以在 Rancher 中重定向服务请求，并了解负载均衡器的限制 +--- + + + + + +在 Rancher 中，你可以通过设置负载均衡器和 Ingress Controller 来重定向服务请求。 + +## 负载均衡器 + +启动应用程序后，该应用程序仅在集群内可用。你无法从集群外部访问它。 + +如果你希望从外部访问应用程序，则必须向集群添加负载均衡器或 Ingress。如果用户知道负载均衡器的 IP 地址和应用的端口号，负载均衡器可以为外部连接创建一个访问集群的网关。 + +Rancher 支持两种类型的负载均衡器： + +- [Layer-4 负载均衡器](layer-4-and-layer-7-load-balancing.md#四层负载均衡器) +- [Layer-7 负载均衡器](layer-4-and-layer-7-load-balancing.md#七层负载均衡器) + +有关详细信息，请参阅[负载均衡器](layer-4-and-layer-7-load-balancing.md)。 + +### 负载均衡器限制 + +负载均衡器有几个需要注意的限制： + +- 负载均衡器只能处理每个 service 的一个 IP 地址。换言之，如果你在集群中运行了多个 service，则必须为每个 service 配备一个负载均衡器。运行多个负载均衡器的花费可能非常高昂。 + +- 如果你想将负载均衡器与托管的 Kubernetes 集群（即托管在 GKE、EKS 或 AKS 中的集群）一起使用，则负载均衡器必须运行在该云提供商的基础设施上。请根据你配置集群的方式查看负载均衡器的兼容列表： + +- [支持 Layer-4 负载均衡](layer-4-and-layer-7-load-balancing.md#四层负载均衡支持) + +- [支持 Layer-7 负载均衡](layer-4-and-layer-7-load-balancing.md#七层负载均衡支持) + +## Ingress + +如上所述，使用负载均衡器的缺点是： + +- 每个服务负载均衡器只能处理一个 IP 地址。 +- 如果你在集群中运行多个服务，则每个服务都必须配备一个负载均衡器。 +- 为每个服务配备负载均衡器的花费可能非常高昂。 + +相反，如果将某个 Ingress 用作集群的入口点，Ingress 可以更灵活地将流量路由到多个 service。它可以将多个 HTTP 请求映射到 service，而无需为每个 service 提供单独的 IP 地址。 + +因此，如果你需要使用相同的 IP 地址、Layer 7 协议或特权节点端口（80 和 443）来公开多个 service，你可以使用一个 Ingress。 + +Ingress 与一个或多个 Ingress Controller 一起动态路由 service 的请求。Ingress 收到请求时，集群中的 Ingress Controller 会根据你配置的 service 子域或路径规则将请求定向到正确的 service。 + +每个 Kubernetes Ingress 资源都对应一个 `/etc/nginx/sites-available/` 中的文件，其中包含一个配置对特定文件和文件夹的请求的 `server{}` 配置块。 + +Ingress 能为你的集群创建一个入口端口（与负载均衡器类似），可以位于集群的内部或外部。RKE 启动的集群中的 Ingress 和 Ingress Controller 由 [Nginx](https://www.nginx.com/) 提供支持。 + +Ingress 还支持其他功能，例如 SSL 终止、基于名称的虚拟主机等。 + +:::note 在高可用性配置中使用 Rancher： + +请避免将 Ingress 添加到 `local` 集群。Rancher 将 Nginx Ingress Controller 作为 Rancher 管理的 _所有_ 集群的全局入口点，其中包括 `local` 集群。因此，当用户尝试访问应用程序时，Rancher 可能会由于重新加载 Nginx 配置而断开连接。要解决这个问题，我们建议你仅在通过 Rancher 启动的集群中部署应用程序。 + +::: + +- 有关如何在 Rancher 中设置 Ingress 的更多信息，请参阅 [Ingress](add-ingresses.md)。 +- 有关 Ingress 和 Ingress Controller 的完整信息，请参阅 [Kubernetes Ingress 文档](https://kubernetes.io/docs/concepts/services-networking/ingress/)。 +- 在项目中使用 Ingress 时，你可以通过设置全局 DNS 条目来将 Ingress 主机名编程到外部 DNS。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md new file mode 100644 index 00000000000..08be0859bb8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md @@ -0,0 +1,82 @@ +--- +title: "Kubernetes 工作负载和 Pod" +description: "了解在 Kubernetes 中构建复杂容器化应用程序的两种结构：Kubernetes 工作负载和 Pod" +--- + + + + + +你可以使用两种基本结构（pod 和工作负载）在 Kubernetes 中构建复杂的容器化应用程序。构建应用程序后，你可以使用第三种结构（service）在集群中或互联网上公开应用程序。 + +### Pod + +[_Pod_](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/) 是一个或多个共享网络命名空间和存储卷的容器。大多数 pod 只有一个容器。因此，我们讨论的 _pod_ 通常等同于 _容器_。扩展 pod 的方式与扩展容器的方式相同，即配置实现服务的同一 pod 的多个实例。通常，Pod 会根据工作负载进行扩展和管理。 + +### 工作负载 + +_工作负载_ 是为 pod 设置部署规则的对象。Kubernetes 基于这些规则执行部署，并根据应用程序的当前状态来更新工作负载。 +工作负载让你可以定义应用程序调度、扩展和升级的规则。 + +#### 工作负载类型 + +Kubernetes 将工作负载分为不同的类型。Kubernetes 支持的最流行的类型是： + +- [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) + + _Deployment_ 最适合用于无状态应用程序（即不需要维护工作负载的状态）。由 Deployment 类型工作负载管理的 Pod 是独立且一次性的。如果 pod 中断了，Kubernetes 会删除该 pod 然后重新创建它。一个示例应用程序是 Nginx Web 服务器。 + +- [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) + + 与 Deployment 相比，_StatefulSet_ 最适合在需要维护身份和存储数据的应用程序中使用。适用的应用程序类似于 Zookeeper（一个需要数据库进行存储的应用程序）。 + +- [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) + + _Daemonset_ 确保集群中的每个节点都运行 pod 的副本。如果你需要收集日志或监控节点性能，这种类似 daemon 的工作负载效果是最好的。 + +- [Job](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) + + _Job_ 启动一个或多个 Pod 并确保指定数量的 Pod 能成功终止。Job 最好用于运行有限任务至完成状态，而不是管理正在进行的应用程序的所需状态。 + +- [CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) + + _CronJobs_ 与 Job 类似。但是，CronJob 会基于 cron 的计划运行到完成状态。 + +### Services + +在许多用例中，工作负载必须： + +- 由集群中的其他工作负载访问。 +- 暴露给外部。 + +你可以通过创建一个 _Service_ 实现这些目的。Service 使用[选择器/标签（查看代码示例）](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#service-and-replicationcontroller)来映射到底层工作负载的 pod。Rancher UI 使用你选择的服务端口和类型来自动创建 service 以及工作负载，从而简化此映射过程。 + +#### Service 类型 + +Rancher 中有几种可用的 Service 类型。以下描述来自 [Kubernetes 文档](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types)。 + +- **ClusterIP** + + > 在集群内部 IP 上公开 Service。如果你选择此值，Service 只能从集群内访问。这是默认的 `ServiceType`。 + +- **NodePort** + + > 在每个节点 IP 上的静态端口（`NodePort`）上暴露 Service。`ClusterIP` service 是自动创建的，而 `NodePort` service 会路由到 ClusterIP service。你可以通过请求 `:` 在集群外部联系 `NodePort` service。 + +- **LoadBalancer** + + > 使用云提供商的负载均衡器向外部公开服务。`NodePort` 和 `ClusterIP` service 是自动创建的，外部负载均衡器会路由到这些 service。 + +## 工作负载选项 + +以下文档介绍了如何部署工作负载和使用工作负载选项。 + +- [部署工作负载](deploy-workloads.md) +- [升级工作负载](upgrade-workloads.md) +- [回滚工作负载](roll-back-workloads.md) + +## 相关链接 + +### 外部链接 + +- [Service](https://kubernetes.io/docs/concepts/services-networking/service/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md new file mode 100644 index 00000000000..9eb5ce53af6 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md @@ -0,0 +1,21 @@ +--- +title: 创建 Nutanix AOS 集群 +description: 使用 Rancher 创建 Nutanix AOS (AHV) 集群。集群可能包括具有不同属性的 VM 组，这些属性可用于细粒度控制节点的大小。 +--- + + + + + +[Nutanix Acropolis 操作系统](https://www.nutanix.com/products/acropolis) (Nutanix AOS) 是适用于 Nutanix 超融合基础架构平台的操作系统。AOS 带有一个名为 [Acropolis Hypervisor（AHV）](https://www.nutanix.com/products/ahv)的内置虚拟机监控程序。你可以结合使用 Rancher 与 Nutanix AOS (AHV)，从而在本地体验云环境的操作。 + +Rancher 可以在 AOS (AHV) 中配置节点并在其上安装 Kubernetes。在 AOS 中创建 Kubernetes 集群时，Rancher 首先与 Prism Central API 通信来配置指定数量的虚拟机。然后在虚拟机上安装 Kubernetes。 + +Nutanix 集群可能由多组具有不同属性（例如内存或 vCPU 数量）的 VM 组成。这种分组允许对每个 Kubernetes 角色的节点大小进行细粒度控制。 + +- [创建 Nutanix 集群](provision-kubernetes-clusters-in-aos.md#creating-a-nutanix-aos-cluster) +- [配置存储](provision-kubernetes-clusters-in-aos.md) + +## 创建 Nutanix 集群 + +在[本节](provision-kubernetes-clusters-in-aos.md)中，你将学习如何使用 Rancher 在 Nutanix AOS 中安装 [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes 集群。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md new file mode 100644 index 00000000000..396f6370108 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md @@ -0,0 +1,60 @@ +--- +title: 创建 vSphere 集群 +description: 使用 Rancher 创建 vSphere 集群。集群可能包括具有不同属性的 VM 组，这些属性可用于细粒度控制节点的大小。 +--- + + + + + +import YouTube from '@site/src/components/YouTube' + +你可以结合使用 Rancher 与 vSphere，从而在本地体验云环境的操作。 + +Rancher 可以在 vSphere 中配置节点并在其上安装 Kubernetes。在 vSphere 中创建 Kubernetes 集群时，Rancher 首先与 vCenter API 通信来配置指定数量的虚拟机。然后在它们之上安装 Kubernetes。 + +vSphere 集群可能由多组具有不同属性（例如内存或 vCPU 数量）的 VM 组成。这种分组允许对每个 Kubernetes 角色的节点大小进行细粒度控制。 + +## Rancher v2.3 中的 vSphere 增强功能 + +我们更新了 vSphere 节点模板，使你可以通过以下增强功能在本地体验云操作： + +### 自我修复的节点池 + +使用 Rancher 配置 vSphere 节点的最大优势之一，是允许你在本地集群中使用 Rancher 的自我修复节点池（也称为[节点自动替换功能](../use-new-nodes-in-an-infra-provider.md#节点自动替换)）。自我修复节点池的功能帮助你替换无状态应用的 worker 节点。当 Rancher 使用节点模板配置节点时，Rancher 可以自动替换无法访问的节点。 + +:::caution + +不建议在 master 节点或连接了持久卷的节点的节点池上启用节点自动替换，因为虚拟机会被临时处理。节点池中的节点与集群断开连接时，其持久卷将被破坏，从而导致有状态应用的数据丢失。 + +::: + +### 实例和调度的动态填充选项 + +vSphere 的节点模板已更新。当你使用 vSphere 凭证创建节点模板时，该模板会自动填充你在 vSphere 控制台中可以访问的相同的虚拟机配置选项。 + +要填充的字段设置需要满足[先决条件](provision-kubernetes-clusters-in-vsphere.md#vsphere-中的准备工作)。 + +### 更多支持的操作系统 + +你可以使用任何支持 `cloud-init` 的操作系统来配置 VM。[cloud config](https://cloudinit.readthedocs.io/en/latest/topics/examples.html) 仅支持 YAML 格式。 + +### v2.3.3 节点模板功能的视频介绍 + +在这段 YouTube 视频中，我们演示了如何使用新的节点模板在本地环境体验云环境一样的操作。 + + + +## 创建 vSphere 集群 + +在[本节](provision-kubernetes-clusters-in-vsphere.md)中，你将学习如何使用 Rancher 在 vSphere 中安装 [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes 集群。 + +## 配置存储 + +有关如何使用 Rancher 在 vSphere 中配置存储的示例，请参阅[本节](../../../manage-clusters/provisioning-storage-examples/vsphere-storage.md)。要在 vSphere 中动态配置存储，你必须启用 vSphere 云提供商。请参阅[树内 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md)和[树外 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 + +## 启用 vSphere 云提供商 + +在 Rancher 中设置云提供商时，Rancher Server 可以自动为集群配置新的基础设施，包括新节点或持久存储设备。 + +有关启用 vSphere 云提供商的详细信息，请参阅[树内 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md)和[树外 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/advanced-user-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/advanced-user-guides.md new file mode 100644 index 00000000000..80d6b10f321 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/advanced-user-guides.md @@ -0,0 +1,11 @@ +--- +title: 高级用户指南 +--- + + + + + +高级用户指南是“问题导向”的文档，用户可以从中学习如何解决问题。高级用于指南与新用户指南的主要区别在于，高级用户指南面向更有经验或更高级的用户，这些用户对文档有更多的技术需求，而且已经了解 Rancher 及其功能。他们知道自己需要做什么，只是需要额外的指导来完成更复杂的任务。 + +应该注意的是，新用户指南和高级用户指南都没有提供详细的解释或讨论（这些文档不包括在本部分）。操作指南侧重于引导用户通过可重复、有效的步骤来学习新技能、掌握某些操作或解决某些问题。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md new file mode 100644 index 00000000000..ce57b2db349 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md @@ -0,0 +1,50 @@ +--- +title: 项目资源配额 +--- + + + + + +如果多个团队共享一个集群，某个团队可能会使用过多的可用资源，例如 CPU、内存、存储、服务、Kubernetes 对象（如 Pod 或 Secret）等。你可以应用 _资源配额_ 来防止过度消耗资源。资源配额是 Rancher 用来限制项目或命名空间可用资源的功能。 + +本文介绍如何在现有项目中创建资源配额。 + +你也可以在创建新项目时设置资源配额。有关详细信息，请参阅[创建新项目](../../../new-user-guides/manage-clusters/projects-and-namespaces.md#创建项目)。 + +Rancher 中的资源配额包含与 [Kubernetes 原生版本](https://kubernetes.io/docs/concepts/policy/resource-quotas/)相同的功能。Rancher 还扩展了资源配额的功能，从而让你将资源配额应用于项目。有关资源配额如何与 Rancher 中的项目一起使用的详细信息，请参阅[此页面](about-project-resource-quotas.md)。 + +### 将资源配额应用于现有项目 + +修改资源配额的使用场景如下： + +- 限制某个项目和项目下的命名空间能使用的资源 +- 在资源配额已生效的情况下，对项目可用的资源进行扩容或缩容 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面，进入要应用资源配额的集群，然后单击 **Explore**。 +1. 单击**集群 > 项目/命名空间**。 +1. 确保 **Projects/Namespaces** 页面处于 **Group by Project** 视图模式。 + ![Screenshot highlighting the "Group by Project" icon, above the list of projects. It resembles a folder.](/img/edit-project-config-for-resource-quotas-group-by-project.png) + +1. 找到要添加资源配额的项目，选择与项目名称同行的 **⋮**。 + ![Screenshot highlighting triple dots icon at the end of the same row as the project name.](/img/edit-project-config-for-resource-quotas-dots.png) + +1. 选择**编辑配置**。 + +1. 展开**资源限额**并单击**添加资源**。你也可以编辑现有配额。 + +1. 选择资源类型。有关类型的更多信息，请参阅[配额类型参考](resource-quota-types.md)。 + +1. 输入**项目限制**和**命名空间默认限制**的值。 + + | 字段 | 描述 | + | ----------------------- | -------------------------------------------------------------------------------------------------------- | + | 项目限制 | 项目的总资源限制。 | + | 命名空间默认限制 | 每个命名空间的默认资源限制。此限制会沿用到项目中的每个命名空间。项目中所有命名空间的限制之和不应超过项目限制。 | + +1. **可选**：添加更多配额。 + +1. 单击**创建**。 + +**结果**：资源配额已应用到你的项目和命名空间。如果你后续需要添加更多命名空间，Rancher 会验证项目是否可以容纳该命名空间。如果项目无法分配资源，你仍然可以创建命名空间，但命名空间将获得的资源配额为 0。然后 Rancher 将不允许你创建任何受此配额限制的资源。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md new file mode 100644 index 00000000000..1ce6b33f141 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md @@ -0,0 +1,41 @@ +--- +title: 项目管理 +--- + + + + + +_项目_ 是 Rancher 中引入的对象，可帮助你更有组织地管理 Kubernetes 集群中的命名空间。你可以使用项目创建多租户集群，这种集群允许一组用户共享相同的底层资源来创建应用，而应用之间不会相互影响。 + +在层次结构方面： + +- 集群包含项目 +- 项目包含命名空间 + +在 Rancher 中，你可以使用项目将多个命名空间作为一个实体进行管理。在原生 Kubernetes（没有项目这个概念）中，RBAC 或集群资源等功能被分配给了各个命名空间。如果集群中的多个命名空间需要分配同样的访问权限，分配权限会变得非常繁琐。即使所有命名空间都需要相同的权限，但也无法使用一个操作中将这些权限应用于所有命名空间。你必须重复地将这些权限分配给每个命名空间。 + +而 Rancher 通过引入项目的概念，通过允许你在项目级别应用资源和访问权限。然后，项目中的每个命名空间都会继承这些资源和策略。因此你只需将资源和策略分配给项目即可，不需要将它们分配给每个单独的命名空间。 + +你可以使用项目执行以下操作： + +- [为用户分配一组命名空间的访问权限](../../new-user-guides/add-users-to-projects.md) +- 为用户分配[项目中的特定角色](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色)。角色可以是所有者、成员、只读或[自定义](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/custom-roles.md) +- [设置资源配额](manage-project-resource-quotas/manage-project-resource-quotas.md) +- [管理命名空间](../../new-user-guides/manage-namespaces.md) +- [配置工具](../../../reference-guides/rancher-project-tools.md) +- [配置 Pod 安全策略](manage-pod-security-policies.md) + +### 授权 + +非管理者用户只有在[管理员](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md)、[集群所有者或成员](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#集群角色)或[项目所有者](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色)将非管理员用户添加到项目的**成员**选项卡后，才能获取项目的访问权限。 + +创建项目的人自动成为[项目所有者](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色)。 + +## 在项目之间切换 + +要在项目之间切换，请使用导航栏中的下拉菜单。你也可以直接在导航栏中切换项目： + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面，进入要切换项目的集群然后点击 **Explore**。 +1. 在顶部导航栏中，选择要打开的项目。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md new file mode 100644 index 00000000000..d99c8f91f92 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md @@ -0,0 +1,100 @@ +--- +title: 备份和灾难恢复 +keywords: [rancher 备份还原, rancher 备份与恢复, 备份恢复 rancher, rancher 备份与恢复 rancher] +--- + + + + + +在本节中，你将学习如何创建 Rancher 的备份，如何从备份中恢复 Rancher，以及如何将 Rancher 迁移到新的 Kubernetes 集群。 + +`rancher-backup` operator 可以用来备份和恢复任何 Kubernetes 集群上的 Rancher。这个应用是一个 Helm Chart，可以通过 Rancher 的 **Apps** 页面或使用 Helm CLI 部署。你可以访问[本页面](https://github.com/rancher/charts/tree/release-v2.6/charts/rancher-backup)获取 `rancher-backup` Helm Chart。 + +backup-restore operator 需要安装在 local 集群上，并且只对 Rancher 应用进行备份。备份和恢复操作仅在本地 Kubernetes 集群中执行。 + + +## 备份和恢复 Docker 安装的 Rancher + +对于使用 Docker 安装的 Rancher，请参见[备份](back-up-docker-installed-rancher.md)和[恢复](restore-docker-installed-rancher.md)对 Rancher 进行备份和恢复。 + +## 备份和恢复原理 + +`rancher-backup` operator 引入了三个自定义资源，分别是 Backups、Restores 和 ResourceSets。将以下集群范围的自定义资源定义添加到集群中： + +- `backups.resources.cattle.io` +- `resourcesets.resources.cattle.io` +- `restores.resources.cattle.io` + +ResourceSet 定义了需要备份哪些 Kubernetes 资源。由于备份 Rancher 所需的值是预设的，因此 ResourceSet 无法通过 Rancher UI 进行配置。请不要修改此 ResourceSet。 + +在创建 Backup 自定义资源时，`rancher-backup` operator 调用 `kube-apiserver` 来获取 Backup 自定义资源引用的 ResourceSet（即预设的 `rancher-resource-set`）资源。 + +然后，operator 以 .tar.gz 格式创建备份文件，并将其存储在 Backup 资源中配置的位置。 + +在创建 Restore 自定义资源时，operator 访问 Restore 指定的 tar.gz 备份文件，并从该文件恢复应用。 + +你可以使用 Rancher UI 或 `kubectl apply` 来创建 Backup 和 Restore 自定义资源。 + +:::note + +请参见[此处](migrate-rancher-to-new-cluster.md#2-使用-restore-自定义资源来还原备份)获取在 Rancher 2.6.3 中将现有备份文件恢复到 v1.22 集群的帮助。 + +::: + +## 安装 rancher-backup operator + +你可以使用 Rancher UI 或 Helm CLI 来安装 `rancher-backup` operator。两种安装方法都将 `rancher-backup` Helm Chart 安装在运行 Rancher Server 的 Kubernetes 集群上。它是集群管理员独有的功能，仅适用于 **local** 集群。（*如果你在 Rancher UI 中没有看到 `rancher-backup`，你可能选择了错误的集群。*） + +:::note + +使用 backup-restore-operator 执行恢复后，Fleet 中会出现一个已知问题：用于 clientSecretName 和 helmSecretName 的密文不包含在 Fleet 的 Git 仓库中。请参见[此处](../../../integrations-in-rancher/fleet/overview.md#故障排除)获得解决方法。 + +::: + +### 使用 Rancher UI 安装 rancher-backup + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面上，转到 `local` 集群并单击 **Explore**。 +1. 在左侧导航栏中，单击 **Apps > Charts**。 +1. 点击 **Rancher 备份**。 +1. 单击**安装**。 +1. 可选：配置默认存储位置。如需获取帮助，请参见[配置](../../../reference-guides/backup-restore-configuration/storage-configuration.md)。 +1. 单击**安装**。 + +**结果**：`rancher-backup` operator 已安装。 + +在**集群仪表板**中，你可以看到列在 **Deployments** 下的 `rancher-backup` operator。 + +如果需要在 Rancher 中配置备份应用，在左侧导航栏中单击 **Rancher 备份**。 + +### RBAC + +只有 Rancher 管理员和本地集群的所有者可以： + +* 安装 Chart +* 看到 Backup 和 Restore CRD 的导航链接 +* 通过分别创建 Backup CR 和 Restore CR 执行备份和恢复 +* 列出目前已执行的备份和恢复操作 + +## 备份 Rancher + +备份是通过创建 Backup 自定义资源实现的。如需查看教程，请参见[本页面](back-up-rancher.md)。 + +## 还原 Rancher + +还原是通过创建 Restore 自定义资源实现的。如需查看教程，请参见[本页面](restore-rancher.md)。 + +## 将 Rancher 迁移到新集群 + +你可以按照[这些步骤](migrate-rancher-to-new-cluster.md)执行迁移。 + +## 默认存储位置配置 + +配置一个用户保存备份文件的默认存储位置。有多种选项可以配置，例如将与 S3 兼容的对象存储作为单个备份文件的存储后端，或在安装 `backup-restore-operator` Helm Chart 时选择一个现有的 `StorageClass`。你还可以选择在每次备份时覆盖配置的存储位置，但这仅限于使用与 S3 兼容的对象存储或 Minio 对象存储。 + +如需了解各个选项的配置，请参见[本页面](../../../reference-guides/backup-restore-configuration/storage-configuration.md)。 + +### 示例 YAML 文件：Rancher Backup Helm Chart + +当使用 Helm CLI 安装时，可以使用示例 [values.yaml 文件](../../../reference-guides/backup-restore-configuration/storage-configuration.md#rancher-backup-helm-chart-的示例-valuesyaml) 来配置 `rancher-backup-operator`。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md new file mode 100644 index 00000000000..4d4d465d136 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md @@ -0,0 +1,21 @@ +--- +title: 跨集群部署应用 +--- + + + + + +不同版本的 Rancher 提供了几种不同的方式来部署跨集群应用。 + +## Fleet + +Rancher v2.5 及更高版本使用 Fleet 跨集群部署应用 + +使用 Fleet 的持续交付是大规模的 GitOps。如需更多信息，请参阅 [Fleet](fleet.md)。 + +### 多集群应用 + +在 v2.5 之前的 Rancher 中，多集群应用功能用于跨集群部署应用。多集群应用功能已弃用，但仍可作为旧版功能使用。 + +详情请参阅[此文档](multi-cluster-apps.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md new file mode 100644 index 00000000000..48ecc783bee --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md @@ -0,0 +1,216 @@ +--- +title: Helm Charts 和 Apps +--- + + + + + +在本节中，你将学习如何在 Rancher 中管理 Helm Chart 仓库和应用。 + +## Helm Charts 在 Rancher 中的工作原理 + +在 Rancher 中 Helm chart 仓库是使用 **Apps** 进行管理的。 + +Rancher 使用应用商店系统导入一系列的 charts 包到仓库里，然后使用这些 charts 来部署自定义的 Kubernetes 应用程序或 Rancher 的工具，如监控（Monitoring）或 Istio。Rancher 工具预先集成进了仓库，并通过独立的 Helm chart 进行部署。如果你有自己的额外仓库只需要添加到当前的集群中就可以正常部署。 + +### Catalogs, Apps, Rancher UI + +[在 Rancher v2.4 及更早版本中](/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md), 存储部署的应用程序的仓库被称为 "catalogs"。这些仓库是通过 UI 的 **Catalogs** 进行管理的。 + +Rancher v2.5 用新的 **Apps & Marketplace** 功能替代了之前的 **应用商店**。 + +从 Rancher v2.6.5 开始, **Apps & Marketplace** 功能在 UI 中被命名为 **Apps**。 + +### Versioning Scheme + +Rancher chart 的版本设计方案主要围绕 charts 的主要版本以及与上游 charts 相关的 `+up` 注释进行设计。 + +**主要版本:** charts 的主要版本与 Rancher 的特定次要版本相关联。当你升级到新的 Rancher 次要版本时，你应该确保所有的功能 charts 也被升级对应的主版本。 + +**基于上游的 Charts:** 在升级时，确保上游 chart 的版本与你的 Rancher 版本兼容。chart 的 `+up` 注释指示 Rancher chart 正在跟踪的上游版本。例如，`100.x.x+up16.6.0` 的监控（Monitoring）跟踪上游的 kube-prometheus-stack `16.6.0` 并附带一些额外的 Rancher 补丁。 + +在升级 Rancher 版本时，不要降级你正在使用的 chart 版本。例如，如果你在 Rancher v2.5 中使用的监控版本比 `16.6.0` 还要高，你不应该升级到 `100.x.x+up16.6.0`，而是应该在下一个发布中升级到适当的版本。 + +#### 预发布版本 + +预发布版本遵循[语义化版本 2.0.0 ](https://semver.org/)的[规范](https://semver.org/#spec-item-9)。 例如，版本为 `0.1.3-dev.12ab4f` 的 Helm chart 被认为是一个预发布版本。预发布版本默认不显示，必须进行配置以显示。 + +如何显示预发布版本： + +1. 点击右上角的用户头像。 +2. 点击 **Preferences**. +3. 在 **Helm Charts** 下, 选择 **Include Prerelease Versions**. + +### 版本控制方案 + +| **Name** | **支持的最低版本** | **支持的最高版本** | +| ---------------- | ------------ | ------------ | +| external-ip-webhook | 100.0.0+up1.0.0 | 100.0.1+up1.0.1 | +| harvester-cloud-provider | 100.0.2+up0.1.12 | 100.0.2+up0.1.12 | +| harvester-csi-driver | 100.0.2+up0.1.11 | 100.0.2+up0.1.11 | +| neuvector | 100.0.0+up2.2.0 | 100.0.0+up2.2.0 | +| rancher-alerting-drivers | 100.0.0 | 100.0.2 | +| rancher-backup | 2.0.1 | 2.1.2 | +| rancher-cis-benchmark | 2.0.1 | 2.0.4 | +| rancher-gatekeeper | 100.0.0+up3.6.0 | 100.1.0+up3.7.1 | +| rancher-istio | 100.0.0+up1.10.4 | 100.3.0+up1.13.3 | +| rancher-logging | 100.0.0+up3.12.0 | 100.1.2+up3.17.4 | +| rancher-longhorn | 100.0.0+up1.1.2 | 100.1.2+up1.2.4 | +| rancher-monitoring | 100.0.0+up16.6.0 | 100.1.2+up19.0.3 | +| rancher-sriov (experimental) | 100.0.0+up0.1.0 | 100.0.3+up0.1.0 | +| rancher-vsphere-cpi | 100.3.0+up1.2.1 | 100.3.0+up1.2.1 | +| rancher-vsphere-csi | 100.3.0+up2.5.1-rancher1 | 100.3.0+up2.5.1-rancher1 | +| rancher-wins-upgrader | 0.0.100 | 100.0.1+up0.0.1 | + +## 访问 Charts + +**Charts** 页面包含所有 Rancher、Rancher 合作伙伴和自定义 Chart。你可以通过选择左侧的下拉菜单来筛选 Chart： + +* Rancher 工具（例如 Logging 或 Monitoring）包含在 **Rancher** 标签下 +* Partner Chart 位于 **Partner** 标签下 +* 自定义 Chart 将显示在仓库的名称下 + +所有这三种类型都以相同的方式部署和管理。 + +:::note +由 Cluster Manager （旧版 Rancher UI 中的全局视图）管理的应用应继续仅由 Cluster Manager 管理，而在新 UI 中使用 Apps 管理的应用则仅能由 Apps 管理。 +::: + +访问 **Charts** 页面: + +1. 点击 **☰ > Cluster Management**。 +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中, 点击 **Apps > Charts**。 + +### 管理仓库 + +**Repositories** 页面列出了你的 Helm 仓库。 这包括传统的具有 index.yaml 的 Helm 端点，以及克隆并指向特定分支的 Git 仓库。要使用自定义 Charts，在这里可以添加你的仓库。添加仓库后，你可以在 **Charts** 页面中访问自定义 Charts，这些 Charts 将列在仓库的名称下。 + +访问 **Repositories** 页面: + +1. 点击 **☰ > Cluster Management**. +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 + +### 添加自定义 Git 仓库 + +要添加一个包含你的 Helm Charts 或集群模板定义的自定义 Git 仓库： + +1. 点击 **☰ > Cluster Management**。 +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 +4. 点击 **Create**。 +5. 选择 **Git repository containing Helm chart...**。 +6. 你必须输入名称和 Git 仓库的 URL。其他配置项包括描述，都是可选的。如果你不想设置默认的分支，可以输入你想要使用的分支名称。通常，默认分支名为 `main` 或 `master`。 +7. 点击 **Create** 添加。 + +在 Rancher 中添加 Charts 仓库后，它将立即生效。 + +### 添加自定义 Helm Chart 仓库 + +你可以将自己的 Helm chart 仓库添加到 Rancher。为了能正确添加 http 的 Helm Chart 仓库，你需要提供 chart 的服务器并能够响应 GET 请求并提供 YAML 文件和 tar 包。 + +有关 Helm chart 仓库的更多信息，请参阅 [官方 Helm 文档](https://helm.sh/docs/topics/chart_repository/)。 + +要将自定义 Helm chart 仓库添加到 Rancher： + +1. 点击 **☰ > Cluster Management**。 +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 +4. 点击 **Create**。 +5. 选择 **http(s) URL to an index generated by Helm**. +6. 输入仓库名称和 chart 的 index URL 地址。 +7. 点击 **Create** 添加。 + +### 添加私有 Git/Helm Chart 仓库 + +你可以使用 SSH 密钥凭据或 HTTP 基础认证秘密（如用户名和密码）添加私有 Git 或 Helm chart 仓库。 + +### 向仓库添加私有 CA + +向 Helm chart 仓库添加私有 CA，你必须将 DER 格式的 CA 证书的 base64 编码副本添加到 Chart 仓库的 `spec.caBundle 字段`，例如 `openssl x509 -outform der -in ca.pem | base64 -w0`。无论是基于 Git 还是 HTTP 的仓库，操作步骤都是相同的 + +1. 点击 **☰**。在左侧导航菜单的 **Explore Cluster**, 选择一个集群。 +2. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 +3. 找到你想要向其添加私有 CA 证书的 Git 或 HTTP 的仓库。点击 **⋮ > Edit YAML**。 +4. 设置 `caBundle` 值，如以下示例： + + ```yaml + [...] + spec: + caBundle: + MIIFXzCCA0egAwIBAgIUWNy8WrvSkgNzV0zdWRP79j9cVcEwDQYJKoZIhvcNAQELBQAwPzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRQwEgYDVQQKDAtNeU9yZywgSW5jLjENMAsGA1UEAwwEcm9vdDAeFw0yMTEyMTQwODMyMTdaFw0yNDEwMDMwODMyMT + ... + nDxZ/tNXt/WPJr/PgEB3hQdInDWYMg7vGO0Oz00G5kWg0sJ0ZTSoA10ZwdjIdGEeKlj1NlPyAqpQ+uDnmx6DW+zqfYtLnc/g6GuLLVPamraqN+gyU8CHwAWPNjZonFN9Vpg0PIk1I2zuOc4EHifoTAXSpnjfzfyAxCaZsnTptimlPFJJqAMj+FfDArGmr4= + [...] + ``` + +:::note 带有认证的 Helm chart 仓库 + +Repo.Spec 包含一个 `disableSameOriginCheck` 值，该值允许用户绕过相同源的检查，将仓库身份认证信息作为基本 Auth 标头与所有 API 调用一起发送。不建议采用这种做法，但这可以用作非标准 Helm Chart 仓库（例如重定向到不同源 URL 的仓库）的临时解决方案。 + +要将此功能用于现有 Helm Chart 仓库，请按照前面的步骤编辑 YAML。在 YAML 文件的 `spec` 部分，添加 `disableSameOriginCheck` 并将其设置为 `true`： + +```yaml +[...] +spec: + disableSameOriginCheck: true +[...] +``` + +::: + +### Helm 兼容性 + +仅支持 Helm 3 兼容 Chart 。 + +### 部署和升级 Chart + +安装和部署 chart: + +1. 点击 **☰ > Cluster Management**。 +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Charts**。 +4. 选择一个 chart 点击 **Install**。 + +Rancher 和 Partner Chart 可能通过自定义页面或 questions.yaml 文件进行额外的配置，但所有 Chart 安装都可以修改 values.yaml 和其他基本设置。单击安装后，将部署一个 Helm 操作作业，并显示该作业的控制台。 + +要查看所有最近的更改，点击左侧导航栏菜单中的 **APPs > Recent Operations**。你可以查看已进行的调用、条件、事件和日志 + +安装 Chart 后，点击左侧导航栏菜单中的 **Apps > Installed Apps**。在本节中，你可以升级或删除安装，并查看更多详细信息。选择升级时，呈现的形式和数值与安装相同。 + +大多数 Rancher 工具在 **Apps** 下方的工具栏中都有额外的页面，以帮助你管理和使用这些功能。这些页面包括指向仪表板的链接、可轻松添加自定义资源的表单以及其他信息。 + +:::caution + +如果在升级前使**自定义 Helm 选项**，并且你的 Chart 中包含不可更改的字段，使用 `--force` 选项可能会导致错误。这是因为 Kubernetes 中的某些对象一旦创建就无法更改。要避免该错误，你可以： + +* 使用默认升级选项（即不要使用 `--force` 选项） +* 卸载现有 Chart 并安装升级后的 Chart +* 在执行强制升级之前删除集群中具有不可更改字段的资源 + +::: + +#### 旧版应用 + +在 **Apps > Installed Apps** 页面上，旧版应用没有升级按钮。 + +如果你想升级已安装的旧版应用， 必须启用[旧版功能](../../advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 如果你在升级 Rancher 之前已经运行了旧版应用，此标志会自动启用。 + +1. 如果尚未启用，启用[旧版功能](../../advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 +2. 点击 **☰ > Cluster Management**。 +3. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +4. 点击 **Legacy > Project > Apps**. + +如果在 **Legacy > Project** 下看不到 **Apps** ,请点击顶部导航栏的 **project/namespace** 搜索栏，并从下拉菜单中选择相关项目。 + +要升级旧版多集群应用： + +1. 点击 **☰**. +2. 在 **Legacy Apps** 下点击 **Multi-cluster Apps**. + + +### 限制 + +Rancher CLI 不能用于安装仪表板应用程序或 Rancher 功能 Chart。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md new file mode 100644 index 00000000000..b50aecb39a9 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md @@ -0,0 +1,30 @@ +--- +title: Horizontal Pod Autoscaler +description: 了解 Pod 水平自动扩缩 (HPA)。如何管理 HPA 以及如何使用服务部署来进行测试 +--- + + + + + +[Horizontal Pod Autoscaler（HPA）](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)是一项 Kubernetes 功能，用于将集群配置为自动扩缩其运行的服务。 + +Rancher 提供了一些附加功能来帮助管理 HPA，具体取决于 Rancher 的版本。 + +你可以使用 Rancher UI 创建、管理和删除 HPA。仅在 `autoscaling/v2beta2` API 中支持 HPA。 + +## 管理 HPA + +管理 HPA 的方式因你的 Kubernetes API 版本而异： + +- **Kubernetes API 版本 autoscaling/V2beta1**：允许根据应用程序的 CPU 和内存利用率自动扩缩 pod。 +- **Kubernetes API 版本 autoscaling/V2beta2**：允许根据 CPU 和内存利用率以及自定义指标自动扩缩 pod。 + +你可以使用 Rancher UI 创建、管理和删除 HPA。在 Rancher UI 中，你可以将 HPA 配置为根据 CPU 和内存利用率进行扩缩。有关详细信息，请参阅[使用 Rancher UI 管理 HPA](manage-hpas-with-ui.md)。如需根据自定义指标进行 HPA，你仍然需要使用 `kubectl`。有关详细信息，请参阅[配置 HPA 以使用 Prometheus 自定义指标进行扩缩](manage-hpas-with-kubectl.md#使用资源指标cpu-和内存配置-hpa-以进行扩缩)。 + +在 Rancher 2.0.7 及更高版本中创建的集群自动满足使用 HPA 的所有要求（metrics-server 和 Kubernetes 集群配置）。 +## 使用服务部署测试 HPA + +你可以转到你的项目并单击 **资源 > HPA** 来查看​​ HPA 当前的副本数。有关详细信息，请参阅[获取 HPA 指标和状态](manage-hpas-with-ui.md)。 + +你还可以使用 `kubectl` 来获取你使用负载测试工具测试的 HPA 的状态。有关详细信息，请参阅[使用 kubectl 测试 HPA](test-hpas-with-kubectl.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md new file mode 100644 index 00000000000..4dd86bb2a6c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md @@ -0,0 +1,67 @@ +--- +title: Kubernetes 资源 +--- + + + + + +你可以在 Rancher UI 中查看和操作 Kubernetes 集群中的所有自定义资源和 CRD。 + +## 工作负载 + +使用[工作负载](workloads-and-pods/workloads-and-pods.md)将应用部署到集群节点，工作负载是包含用于运行应用的 pod 的对象，以及为部署行为设置规则的元数据。工作负载可以部署在集群范围内，也可以部署在一个命名空间内。 + +部署工作负载时，你可以使用任何镜像进行部署。可供选择的[工作负载类型](workloads-and-pods/workloads-and-pods.md#工作负载类型)有多种，工作负载类型决定了你的应用程序的运行方式。 + +在工作负载部署之后，你可以继续使用它。你可以： + +- 将工作负载[升级](workloads-and-pods/upgrade-workloads.md)到它运行的应用的更新版本。 +- 如果升级出现问题，将工作负载[回滚](workloads-and-pods/roll-back-workloads.md)到以前的版本。 +- [添加一个 sidecar](workloads-and-pods/add-a-sidecar.md)，这是一个支持主要工作负载的工作负载。 + +## 负载均衡和 Ingress + +### 负载均衡器 + +启动应用程序后，它仅在集群中可用。无法从外部访问它。 + +如果你希望你的应用程序可以从外部访问，则必须向集群添加负载均衡器。如果用户知道负载均衡器的 IP 地址和应用的端口号，负载均衡器可以为外部连接创建一个访问集群的网关。 + +Rancher 支持两种类型的负载均衡器： + +- [Layer-4 负载均衡器](load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing.md#layer-4-load-balancer#四层负载均衡器) +- [Layer-7 负载均衡器](load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing.md#七层负载均衡器) + +有关详细信息，请参阅[负载均衡器](load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing.md)。 + +#### Ingress + +负载均衡器只能处理每个 service 的一个 IP 地址。换言之，如果你在集群中运行了多个 service，则必须为每个 service 配备一个负载均衡器。运行多个负载均衡器的花费可能非常高昂。因此，你可以使用 Ingress 来解决此问题。 + +Ingress 是一组充当负载均衡器的规则。Ingress 与一个或多个 Ingress Controller 一起动态路由 service 的请求。Ingress 收到请求时，集群中的 Ingress Controller 会对负载均衡器进行配置，从而根据你配置的 service 子域或路径规则将请求定向到正确的 service。 + +有关详细信息，请参阅 [Ingress](load-balancer-and-ingress-controller/add-ingresses.md)。 + +在项目中使用 Ingress 时，你可以通过设置全局 DNS 条目来将 Ingress 主机名编程到外部 DNS。 + +## 服务发现 + +使用负载均衡器和/或 Ingress 将集群公开给外部请求后，你只能通过 IP 地址访问集群。要创建可解析的主机名，你必须创建服务记录，该记录将 IP 地址、外部主机名、DNS 记录别名、工作负载或标记的 pod 映射到特定主机名。 + +有关详细信息，请参阅[服务发现](create-services.md)。 + +## 应用程序 + +除了启动应用程序的各个组件外，你还可以使用 Rancher 应用商店来启动应用，即 Helm Chart。 + +## Kubernetes 资源 + +在 Rancher 项目或命名空间的上下文中，_资源_ 是支持 Pod 操作的文件和数据。在 Rancher 中，证书、镜像仓库和密文都被视为资源。但是，Kubernetes 将资源划分为不同类型的[密文（secret）](https://kubernetes.io/docs/concepts/configuration/secret/)。因此，在单个项目或命名空间中，各个资源必须具有唯一的名称以避免冲突。资源主要用于承载敏感信息，但也有其他用途。 + +资源包括： + +- [证书](encrypt-http-communication.md)：用于加密/解密进入或离开集群的数据的文件。 +- [ConfigMap](configmaps.md)：存储一般配置信息的文件，例如一组配置文件。 +- [密文](secrets.md)：存储密码、token 或密钥等敏感数据的文件。 +- [镜像仓库](kubernetes-and-docker-registries.md)：携带用于验证私有镜像仓库的凭证的文件。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md new file mode 100644 index 00000000000..00ae5a04894 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md @@ -0,0 +1,65 @@ +--- +title: 在 Rancher 中设置负载均衡器和 Ingress Controller +description: 了解如何设置负载均衡器和 Ingress Controller 以在 Rancher 中重定向服务请求，并了解负载均衡器的限制 +--- + + + + + +在 Rancher 中，你可以通过设置负载均衡器和 Ingress Controller 来重定向服务请求。 + +## 负载均衡器 + +启动应用程序后，该应用程序仅在集群内可用。你无法从集群外部访问它。 + +如果你希望从外部访问应用程序，则必须向集群添加负载均衡器或 Ingress。如果用户知道负载均衡器的 IP 地址和应用的端口号，负载均衡器可以为外部连接创建一个访问集群的网关。 + +Rancher 支持两种类型的负载均衡器： + +- [Layer-4 负载均衡器](layer-4-and-layer-7-load-balancing.md#四层负载均衡器) +- [Layer-7 负载均衡器](layer-4-and-layer-7-load-balancing.md#七层负载均衡器) + +有关详细信息，请参阅[负载均衡器](layer-4-and-layer-7-load-balancing.md)。 + +### 负载均衡器限制 + +负载均衡器有几个需要注意的限制： + +- 负载均衡器只能处理每个 service 的一个 IP 地址。换言之，如果你在集群中运行了多个 service，则必须为每个 service 配备一个负载均衡器。运行多个负载均衡器的花费可能非常高昂。 + +- 如果你想将负载均衡器与托管的 Kubernetes 集群（即托管在 GKE、EKS 或 AKS 中的集群）一起使用，则负载均衡器必须运行在该云提供商的基础设施上。请根据你配置集群的方式查看负载均衡器的兼容列表： + +- [支持 Layer-4 负载均衡](layer-4-and-layer-7-load-balancing.md#四层负载均衡支持) + +- [支持 Layer-7 负载均衡](layer-4-and-layer-7-load-balancing.md#七层负载均衡支持) + +## Ingress + +如上所述，使用负载均衡器的缺点是： + +- 每个服务负载均衡器只能处理一个 IP 地址。 +- 如果你在集群中运行多个服务，则每个服务都必须配备一个负载均衡器。 +- 为每个服务配备负载均衡器的花费可能非常高昂。 + +相反，如果将某个 Ingress 用作集群的入口点，Ingress 可以更灵活地将流量路由到多个 service。它可以将多个 HTTP 请求映射到 service，而无需为每个 service 提供单独的 IP 地址。 + +因此，如果你需要使用相同的 IP 地址、Layer 7 协议或特权节点端口（80 和 443）来公开多个 service，你可以使用一个 Ingress。 + +Ingress 与一个或多个 Ingress Controller 一起动态路由 service 的请求。Ingress 收到请求时，集群中的 Ingress Controller 会根据你配置的 service 子域或路径规则将请求定向到正确的 service。 + +每个 Kubernetes Ingress 资源都对应一个 `/etc/nginx/sites-available/` 中的文件，其中包含一个配置对特定文件和文件夹的请求的 `server{}` 配置块。 + +Ingress 能为你的集群创建一个入口端口（与负载均衡器类似），可以位于集群的内部或外部。RKE 启动的集群中的 Ingress 和 Ingress Controller 由 [Nginx](https://www.nginx.com/) 提供支持。 + +Ingress 还支持其他功能，例如 SSL 终止、基于名称的虚拟主机等。 + +:::note 在高可用性配置中使用 Rancher： + +请避免将 Ingress 添加到 `local` 集群。Rancher 将 Nginx Ingress Controller 作为 Rancher 管理的 _所有_ 集群的全局入口点，其中包括 `local` 集群。因此，当用户尝试访问应用程序时，Rancher 可能会由于重新加载 Nginx 配置而断开连接。要解决这个问题，我们建议你仅在通过 Rancher 启动的集群中部署应用程序。 + +::: + +- 有关如何在 Rancher 中设置 Ingress 的更多信息，请参阅 [Ingress](add-ingresses.md)。 +- 有关 Ingress 和 Ingress Controller 的完整信息，请参阅 [Kubernetes Ingress 文档](https://kubernetes.io/docs/concepts/services-networking/ingress/)。 +- 在项目中使用 Ingress 时，你可以通过设置全局 DNS 条目来将 Ingress 主机名编程到外部 DNS。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md new file mode 100644 index 00000000000..08be0859bb8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md @@ -0,0 +1,82 @@ +--- +title: "Kubernetes 工作负载和 Pod" +description: "了解在 Kubernetes 中构建复杂容器化应用程序的两种结构：Kubernetes 工作负载和 Pod" +--- + + + + + +你可以使用两种基本结构（pod 和工作负载）在 Kubernetes 中构建复杂的容器化应用程序。构建应用程序后，你可以使用第三种结构（service）在集群中或互联网上公开应用程序。 + +### Pod + +[_Pod_](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/) 是一个或多个共享网络命名空间和存储卷的容器。大多数 pod 只有一个容器。因此，我们讨论的 _pod_ 通常等同于 _容器_。扩展 pod 的方式与扩展容器的方式相同，即配置实现服务的同一 pod 的多个实例。通常，Pod 会根据工作负载进行扩展和管理。 + +### 工作负载 + +_工作负载_ 是为 pod 设置部署规则的对象。Kubernetes 基于这些规则执行部署，并根据应用程序的当前状态来更新工作负载。 +工作负载让你可以定义应用程序调度、扩展和升级的规则。 + +#### 工作负载类型 + +Kubernetes 将工作负载分为不同的类型。Kubernetes 支持的最流行的类型是： + +- [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) + + _Deployment_ 最适合用于无状态应用程序（即不需要维护工作负载的状态）。由 Deployment 类型工作负载管理的 Pod 是独立且一次性的。如果 pod 中断了，Kubernetes 会删除该 pod 然后重新创建它。一个示例应用程序是 Nginx Web 服务器。 + +- [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) + + 与 Deployment 相比，_StatefulSet_ 最适合在需要维护身份和存储数据的应用程序中使用。适用的应用程序类似于 Zookeeper（一个需要数据库进行存储的应用程序）。 + +- [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) + + _Daemonset_ 确保集群中的每个节点都运行 pod 的副本。如果你需要收集日志或监控节点性能，这种类似 daemon 的工作负载效果是最好的。 + +- [Job](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) + + _Job_ 启动一个或多个 Pod 并确保指定数量的 Pod 能成功终止。Job 最好用于运行有限任务至完成状态，而不是管理正在进行的应用程序的所需状态。 + +- [CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) + + _CronJobs_ 与 Job 类似。但是，CronJob 会基于 cron 的计划运行到完成状态。 + +### Services + +在许多用例中，工作负载必须： + +- 由集群中的其他工作负载访问。 +- 暴露给外部。 + +你可以通过创建一个 _Service_ 实现这些目的。Service 使用[选择器/标签（查看代码示例）](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#service-and-replicationcontroller)来映射到底层工作负载的 pod。Rancher UI 使用你选择的服务端口和类型来自动创建 service 以及工作负载，从而简化此映射过程。 + +#### Service 类型 + +Rancher 中有几种可用的 Service 类型。以下描述来自 [Kubernetes 文档](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types)。 + +- **ClusterIP** + + > 在集群内部 IP 上公开 Service。如果你选择此值，Service 只能从集群内访问。这是默认的 `ServiceType`。 + +- **NodePort** + + > 在每个节点 IP 上的静态端口（`NodePort`）上暴露 Service。`ClusterIP` service 是自动创建的，而 `NodePort` service 会路由到 ClusterIP service。你可以通过请求 `:` 在集群外部联系 `NodePort` service。 + +- **LoadBalancer** + + > 使用云提供商的负载均衡器向外部公开服务。`NodePort` 和 `ClusterIP` service 是自动创建的，外部负载均衡器会路由到这些 service。 + +## 工作负载选项 + +以下文档介绍了如何部署工作负载和使用工作负载选项。 + +- [部署工作负载](deploy-workloads.md) +- [升级工作负载](upgrade-workloads.md) +- [回滚工作负载](roll-back-workloads.md) + +## 相关链接 + +### 外部链接 + +- [Service](https://kubernetes.io/docs/concepts/services-networking/service/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md new file mode 100644 index 00000000000..885bbb38847 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md @@ -0,0 +1,21 @@ +--- +title: 创建 Nutanix AOS 集群 +description: 使用 Rancher 创建 Nutanix AOS (AHV) 集群。集群可能包括具有不同属性的 VM 组，这些属性可用于细粒度控制节点的大小。 +--- + + + + + +[Nutanix Acropolis 操作系统](https://www.nutanix.com/products/acropolis) (Nutanix AOS) 是适用于 Nutanix 超融合基础架构平台的操作系统。AOS 带有一个名为 [Acropolis Hypervisor（AHV）](https://www.nutanix.com/products/ahv)的内置虚拟机监控程序。你可以结合使用 Rancher 与 Nutanix AOS (AHV)，从而在本地体验云环境的操作。 + +Rancher 可以在 AOS (AHV) 中配置节点并在其上安装 Kubernetes。在 AOS 中创建 Kubernetes 集群时，Rancher 首先与 Prism Central API 通信来配置指定数量的虚拟机。然后在虚拟机上安装 Kubernetes。 + +Nutanix 集群可能由多组具有不同属性（例如内存或 vCPU 数量）的 VM 组成。这种分组允许对每个 Kubernetes 角色的节点大小进行细粒度控制。 + +- [创建 Nutanix 集群](provision-kubernetes-clusters-in-aos.md#creating-a-nutanix-aos-cluster) +- [配置存储](provision-kubernetes-clusters-in-aos.md) + +## 创建 Nutanix 集群 + +在[本节](provision-kubernetes-clusters-in-aos.md)中，你将学习如何使用 Rancher 在 Nutanix AOS 中安装 [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes 集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md new file mode 100644 index 00000000000..396f6370108 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md @@ -0,0 +1,60 @@ +--- +title: 创建 vSphere 集群 +description: 使用 Rancher 创建 vSphere 集群。集群可能包括具有不同属性的 VM 组，这些属性可用于细粒度控制节点的大小。 +--- + + + + + +import YouTube from '@site/src/components/YouTube' + +你可以结合使用 Rancher 与 vSphere，从而在本地体验云环境的操作。 + +Rancher 可以在 vSphere 中配置节点并在其上安装 Kubernetes。在 vSphere 中创建 Kubernetes 集群时，Rancher 首先与 vCenter API 通信来配置指定数量的虚拟机。然后在它们之上安装 Kubernetes。 + +vSphere 集群可能由多组具有不同属性（例如内存或 vCPU 数量）的 VM 组成。这种分组允许对每个 Kubernetes 角色的节点大小进行细粒度控制。 + +## Rancher v2.3 中的 vSphere 增强功能 + +我们更新了 vSphere 节点模板，使你可以通过以下增强功能在本地体验云操作： + +### 自我修复的节点池 + +使用 Rancher 配置 vSphere 节点的最大优势之一，是允许你在本地集群中使用 Rancher 的自我修复节点池（也称为[节点自动替换功能](../use-new-nodes-in-an-infra-provider.md#节点自动替换)）。自我修复节点池的功能帮助你替换无状态应用的 worker 节点。当 Rancher 使用节点模板配置节点时，Rancher 可以自动替换无法访问的节点。 + +:::caution + +不建议在 master 节点或连接了持久卷的节点的节点池上启用节点自动替换，因为虚拟机会被临时处理。节点池中的节点与集群断开连接时，其持久卷将被破坏，从而导致有状态应用的数据丢失。 + +::: + +### 实例和调度的动态填充选项 + +vSphere 的节点模板已更新。当你使用 vSphere 凭证创建节点模板时，该模板会自动填充你在 vSphere 控制台中可以访问的相同的虚拟机配置选项。 + +要填充的字段设置需要满足[先决条件](provision-kubernetes-clusters-in-vsphere.md#vsphere-中的准备工作)。 + +### 更多支持的操作系统 + +你可以使用任何支持 `cloud-init` 的操作系统来配置 VM。[cloud config](https://cloudinit.readthedocs.io/en/latest/topics/examples.html) 仅支持 YAML 格式。 + +### v2.3.3 节点模板功能的视频介绍 + +在这段 YouTube 视频中，我们演示了如何使用新的节点模板在本地环境体验云环境一样的操作。 + + + +## 创建 vSphere 集群 + +在[本节](provision-kubernetes-clusters-in-vsphere.md)中，你将学习如何使用 Rancher 在 vSphere 中安装 [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes 集群。 + +## 配置存储 + +有关如何使用 Rancher 在 vSphere 中配置存储的示例，请参阅[本节](../../../manage-clusters/provisioning-storage-examples/vsphere-storage.md)。要在 vSphere 中动态配置存储，你必须启用 vSphere 云提供商。请参阅[树内 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md)和[树外 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 + +## 启用 vSphere 云提供商 + +在 Rancher 中设置云提供商时，Rancher Server 可以自动为集群配置新的基础设施，包括新节点或持久存储设备。 + +有关启用 vSphere 云提供商的详细信息，请参阅[树内 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md)和[树外 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 From 05e29d94de939b43719c1a43ddb1866026850cfe Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 8 Apr 2024 12:26:40 -0700 Subject: [PATCH 23/67] Updating the capi.md page to cluster-api.md to correct intended URL. Signed-off-by: Sunil Singh --- .../cluster-api/{capi.md => cluster-api.md} | 0 sidebars.js | 2 +- .../cluster-api/{capi.md => cluster-api.md} | 0 .../cluster-api/{capi.md => cluster-api.md} | 0 versioned_sidebars/version-2.7-sidebars.json | 2 +- versioned_sidebars/version-2.8-sidebars.json | 2 +- 6 files changed, 3 insertions(+), 3 deletions(-) rename docs/integrations-in-rancher/cluster-api/{capi.md => cluster-api.md} (100%) rename versioned_docs/version-2.7/integrations-in-rancher/cluster-api/{capi.md => cluster-api.md} (100%) rename versioned_docs/version-2.8/integrations-in-rancher/cluster-api/{capi.md => cluster-api.md} (100%) diff --git a/docs/integrations-in-rancher/cluster-api/capi.md b/docs/integrations-in-rancher/cluster-api/cluster-api.md similarity index 100% rename from docs/integrations-in-rancher/cluster-api/capi.md rename to docs/integrations-in-rancher/cluster-api/cluster-api.md diff --git a/sidebars.js b/sidebars.js index ac4e0200b3d..0bde0d57b45 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1220,7 +1220,7 @@ const sidebars = { "label": "Cluster API (CAPI) with Rancher Turtles", "link": { "type": "doc", - "id": "integrations-in-rancher/cluster-api/capi" + "id": "integrations-in-rancher/cluster-api/cluster-api" }, "items": [ "integrations-in-rancher/cluster-api/overview" diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/capi.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/cluster-api.md similarity index 100% rename from versioned_docs/version-2.7/integrations-in-rancher/cluster-api/capi.md rename to versioned_docs/version-2.7/integrations-in-rancher/cluster-api/cluster-api.md diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md similarity index 100% rename from versioned_docs/version-2.8/integrations-in-rancher/cluster-api/capi.md rename to versioned_docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md diff --git a/versioned_sidebars/version-2.7-sidebars.json b/versioned_sidebars/version-2.7-sidebars.json index 5c8ee7ab98b..eb8099a8e4a 100644 --- a/versioned_sidebars/version-2.7-sidebars.json +++ b/versioned_sidebars/version-2.7-sidebars.json @@ -1220,7 +1220,7 @@ "label": "Cluster API (CAPI) with Rancher Turtles", "link": { "type": "doc", - "id": "integrations-in-rancher/cluster-api/capi" + "id": "integrations-in-rancher/cluster-api/cluster-api" }, "items": [ "integrations-in-rancher/cluster-api/overview" diff --git a/versioned_sidebars/version-2.8-sidebars.json b/versioned_sidebars/version-2.8-sidebars.json index 824b2892428..acd367f8d41 100644 --- a/versioned_sidebars/version-2.8-sidebars.json +++ b/versioned_sidebars/version-2.8-sidebars.json @@ -1180,7 +1180,7 @@ "label": "Cluster API (CAPI) with Rancher Turtles", "link": { "type": "doc", - "id": "integrations-in-rancher/cluster-api/capi" + "id": "integrations-in-rancher/cluster-api/cluster-api" }, "items": [ "integrations-in-rancher/cluster-api/overview" From d60a015493b541f9ed9547c9cb0e4ea76daa68a6 Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Mon, 8 Apr 2024 16:32:07 -0400 Subject: [PATCH 24/67] syncing with 1218 (#1222) --- .../rancher-extensions.md | 9 +- .../rancher-extensions.md | 162 ++++++++++++------ 2 files changed, 109 insertions(+), 62 deletions(-) diff --git a/docs/integrations-in-rancher/rancher-extensions.md b/docs/integrations-in-rancher/rancher-extensions.md index 18fa4f2e549..52901b706b3 100644 --- a/docs/integrations-in-rancher/rancher-extensions.md +++ b/docs/integrations-in-rancher/rancher-extensions.md @@ -6,7 +6,7 @@ title: Rancher Extensions -New in Rancher v2.7.0, Rancher introduces **extensions**. Extensions allow users, developers, partners, and customers to extend and enhance the Rancher UI. In addition, users can make changes and create enhancements to their UI functionality independent of Rancher releases. Extensions will enable users to build on top of Rancher to better tailor it to their respective environments. Note that users will also have the ability to update to new versions as well as roll back to a previous version. +Extensions allow users, developers, partners, and customers to extend and enhance the Rancher UI. In addition, users can make changes and create enhancements to their UI functionality independent of Rancher releases. Extensions will enable users to build on top of Rancher to better tailor it to their respective environments. Note that users will also have the ability to update to new versions as well as roll back to a previous version. Extensions are Helm charts that can only be installed once into a cluster; therefore, these charts have been simplified and separated from the general Helm charts listed under **Apps**. @@ -28,13 +28,6 @@ Examples of built-in Rancher extensions are Fleet, Explorer, and Harvester. Exam 3. On the **Extensions** page, click on the **Available** tab to select which extensions you want to install. -:::info - -In v2.7.0, the built-in extensions aren't displayed under the **Available** tab. Therefore, you'll need to manually add the desired repos to install extensions. - -::: -
- 4. If no extensions are showing as available, you may manually add repos as follows: 4.1. On the upper right of screen, click on **⋮ > Manage Repositories > Create**. diff --git a/versioned_docs/version-2.8/integrations-in-rancher/rancher-extensions.md b/versioned_docs/version-2.8/integrations-in-rancher/rancher-extensions.md index 2c56082fc53..52901b706b3 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/rancher-extensions.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/rancher-extensions.md @@ -6,7 +6,7 @@ title: Rancher Extensions -New in Rancher v2.7.0, Rancher introduces **extensions**. Extensions allow users, developers, partners, and customers to extend and enhance the Rancher UI. In addition, users can make changes and create enhancements to their UI functionality independent of Rancher releases. Extensions will enable users to build on top of Rancher to better tailor it to their respective environments. Note that users will also have the ability to update to new versions as well as roll back to a previous version. +Extensions allow users, developers, partners, and customers to extend and enhance the Rancher UI. In addition, users can make changes and create enhancements to their UI functionality independent of Rancher releases. Extensions will enable users to build on top of Rancher to better tailor it to their respective environments. Note that users will also have the ability to update to new versions as well as roll back to a previous version. Extensions are Helm charts that can only be installed once into a cluster; therefore, these charts have been simplified and separated from the general Helm charts listed under **Apps**. @@ -28,13 +28,6 @@ Examples of built-in Rancher extensions are Fleet, Explorer, and Harvester. Exam 3. On the **Extensions** page, click on the **Available** tab to select which extensions you want to install. -:::info - -In v2.7.0, the built-in extensions aren't displayed under the **Available** tab. Therefore, you'll need to manually add the desired repos to install extensions. - -::: -
- 4. If no extensions are showing as available, you may manually add repos as follows: 4.1. On the upper right of screen, click on **⋮ > Manage Repositories > Create**. @@ -53,29 +46,32 @@ In v2.7.0, the built-in extensions aren't displayed under the **Available** tab. ![Reload button](/img/reload-button.png) -### Importing and Installing Extensions in an Air-Gapped Environment +## Updating and Upgrading Extensions -1. Find the address of the container image repository that you want to import as an extension. Rancher provides some extensions, such as Kubewarden and Elemental, through the `ui-plugin-catalog` container image at https://hub.docker.com/r/rancher/ui-plugin-catalog/tags. You should import and use the latest tagged version of the image to ensure you receive the latest features and security updates. +1. Click **☰ > Extensions** under **Configuration**. +1. Select the **Updates** tab. +1. Click **Update**. - * **(Optional)** If the container image is private: [Create](../how-to-guides/new-user-guides/kubernetes-resources-setup/secrets.md) a registry secret within the `cattle-ui-plugin-system` namespace. Enter the domain of the image address in the **Registry Domain Name** field. +If there is a new version of the extension, there will also be an **Update** button visible on the associated card for the extension in the **Available** tab. + +## Deleting Extensions + +1. Click **☰**, then click on the name of your local cluster. +1. From the sidebar, select **Apps > Installed Apps**. +1. Find the name of the chart you want to delete and select the checkbox next to it. +1. Click **Delete**. + +## Deleting Extension Repositories + +1. Click **☰ > Extensions** under **Configuration**. +1. On the top right, click **⋮ > Manage Repositories**. +1. Find the name of the extension repository you want to delete. Select the checkbox next to the repository name, then click **Delete**. + +## Deleting Extension Repository Container Images 1. Click **☰**, then select **Extensions**, under **Configuration**. - 1. On the top right, click **⋮ > Manage Extension Catalogs**. - -1. Select the **Import Extension Catalog** button. - -1. Enter the image address in the **Catalog Image Reference** field. - - * **(Optional)** If the container image is private: Select the secret you just created from the **Pull Secrets** drop-down menu. - -1. Click **Load**. The extension will now be **Pending**. - -1. Return to the **Extensions** page. - -1. Select the **Available** tab, and click the **Reload** button to make sure that the list of extensions is up to date. - -1. Find the extension you just added, and click the **Install** button. +1. Find the name of the container image you want to delete, then click **⋮ > Uninstall**. ## Uninstalling Extensions @@ -95,13 +91,94 @@ You must reload the page after disabling extensions or display issues may occur. ::: -## Updating and Upgrading Extensions +## Developing Extensions -1. Click **☰ > Extensions** under **Configuration**. -1. Select the **Updates** tab. -1. Click **Update**. +To learn how to develop your own extensions, refer to the official [Getting Started](https://rancher.github.io/dashboard/extensions/extensions-getting-started) guide. -If there is a new version of the extension, there will also be an **Update** button visible on the associated card for the extension in the **Available** tab. +## Working with Extensions in an Air-gapped Environment + +If you intend to work with extensions in an air-gapped environment, you must perform some extra steps before you can complete certain tasks. + +### Accessing Rancher UI Extensions in an Air-Gapped Environment + +Rancher provides some extensions, such as Kubewarden and Elemental, through the `ui-plugin-catalog` container image at https://hub.docker.com/r/rancher/ui-plugin-catalog/tags. If you're trying to install these extensions in an air-gapped environment, you must make the `ui-plugin-catalog` image accessible. + +1. Mirror the `ui-plugin-catalog` image to a private registry: + + ```bash + export REGISTRY_ENDPOINT= # e.g. "my-private-registry.com" + docker pull rancher/ui-plugin-catalog: + docker tag rancher/ui-plugin-catalog: $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog: + docker push $REGISTRY_ENDPOINT/rancher/ui-plugin-catalog: +2. Use the mirrored image to create a Kubernetes [deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/): + ```yaml + apiVersion: apps/v1 + kind: Deployment + metadata: + name: ui-plugin-catalog + namespace: cattle-ui-plugin-system + labels: + catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog + spec: + replicas: 1 + selector: + matchLabels: + catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog + template: + metadata: + namespace: cattle-ui-plugin-system + labels: + catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog + spec: + containers: + - name: server + image: /rancher/ui-plugin-catalog: + imagePullPolicy: Always + imagePullSecrets: + - name: + ``` +3. Expose the deployment by creating a [ClusterIP service](https://kubernetes.io/docs/concepts/services-networking/service/#type-clusterip): + ```yaml + apiVersion: v1 + kind: Service + metadata: + name: ui-plugin-catalog-svc + namespace: cattle-ui-plugin-system + spec: + ports: + - name: catalog-svc-port + port: 8080 + protocol: TCP + targetPort: 8080 + selector: + catalog.cattle.io/ui-extensions-catalog-image: ui-plugin-catalog + type: ClusterIP + ``` +4. Create a [ClusterRepo](../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md) that targets the ClusterIP service: + ```yaml + apiVersion: catalog.cattle.io/v1 + kind: ClusterRepo + metadata: + name: ui-plugin-catalog-repo + spec: + url: http://ui-plugin-catalog-svc.cattle-ui-plugin-system:8080 + ``` + +After you successfully set up these resources, you can install the extensions from the `ui-plugin-charts` manifest into your air-gapped environment. + +### Importing and Installing Extensions in an Air-gapped Environment + +1. Find the address of the container image repository that you want to import as an extension. You should import and use the latest tagged version of the image to ensure you receive the latest features and security updates. + - **(Optional)** If the container image is private: [Create](../how-to-guides/new-user-guides/kubernetes-resources-setup/secrets.md) a registry secret within the `cattle-ui-plugin-system` namespace. Enter the domain of the image address in the **Registry Domain Name** field. +1. Click **☰**, then select **Extensions**, under **Configuration**. +1. On the top right, click **⋮ > Manage Extension Catalogs**. +1. Select the **Import Extension Catalog** button. +1. Enter the image address in the **Catalog Image Reference** field. + * **(Optional)** If the container image is private, select the secret you just created from the **Pull Secrets** drop-down menu. +1. Click **Load**. The extension will now be **Pending**. +1. Return to the **Extensions** page. +1. Select the **Available** tab, and click **Reload** to make sure that the list of extensions is up to date. +1. Find the extension you just added, and click **Install**. ### Updating and Upgrading an Extensions Repository in an Air-gapped Environment @@ -119,26 +196,3 @@ After you mirror the latest changes, follow these steps: 1. Click **⋮ > Edit config**. 1. Update the **Container Image** field within the deployment's container with the latest image. 1. Click **Save**. - -## Deleting Helm Charts - -1. Click **☰**, then click on the name of your local cluster. -1. From the sidebar, select **Apps > Installed Apps**. -1. Find the name of the chart you want to delete and select the checkbox next to it. -1. Click **Delete**. - -## Deleting Extension Repositories - -1. Click **☰ > Extensions** under **Configuration**. -1. On the top right, click **⋮ > Manage Repositories**. -1. Find the name of the extension repository you want to delete. Select the checkbox next to the repository name, then click **Delete**. - -## Deleting Extension Repository Container Images - -1. Click **☰**, then select **Extensions**, under **Configuration**. -1. On the top right, click **⋮ > Manage Extension Catalogs**. -1. Find the name of the container image you want to delete. Click **⋮ > Uninstall**. - -## Developing Extensions - -To learn how to develop your own extensions, refer to the official [Getting Started](https://rancher.github.io/dashboard/extensions/extensions-getting-started) guide. From 3d6a25e6ed8b980f463d8d68251c739541487994 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 8 Apr 2024 13:42:20 -0700 Subject: [PATCH 25/67] Fixing markdown links after review. Signed-off-by: Sunil Singh --- docs/integrations-in-rancher/cluster-api/overview.md | 4 ++-- .../integrations-in-rancher/cluster-api/overview.md | 4 ++-- .../integrations-in-rancher/cluster-api/overview.md | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 66887cc8602..7d56e974e75 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -27,7 +27,7 @@ You can install the Rancher Turtles operator via the Rancher UI, or with Helm. T :::caution -If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#installing-via-helm). +If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](#installing-via-helm). ::: @@ -47,7 +47,7 @@ By adding the Turtles repository via the Rancher UI, Rancher can process the ins 1. Click **Rancher Turtles - the Cluster API Extension**. 1. Click **Install > Next > Install**. -This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). +This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). The installation may take a few minutes and after completing you can see the following new deployments in the cluster: diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index 66887cc8602..7d56e974e75 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -27,7 +27,7 @@ You can install the Rancher Turtles operator via the Rancher UI, or with Helm. T :::caution -If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#installing-via-helm). +If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](#installing-via-helm). ::: @@ -47,7 +47,7 @@ By adding the Turtles repository via the Rancher UI, Rancher can process the ins 1. Click **Rancher Turtles - the Cluster API Extension**. 1. Click **Install > Next > Install**. -This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). +This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). The installation may take a few minutes and after completing you can see the following new deployments in the cluster: diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index 66887cc8602..7d56e974e75 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -27,7 +27,7 @@ You can install the Rancher Turtles operator via the Rancher UI, or with Helm. T :::caution -If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](./overview.md#installing-via-helm). +If you already have the Cluster API (CAPI) Operator installed in your cluster, you must use the [manual Helm installation method](#installing-via-helm). ::: @@ -47,7 +47,7 @@ By adding the Turtles repository via the Rancher UI, Rancher can process the ins 1. Click **Rancher Turtles - the Cluster API Extension**. 1. Click **Install > Next > Install**. -This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](./overview.md#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). +This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). The installation may take a few minutes and after completing you can see the following new deployments in the cluster: From 2e834995cdbc11e27a298e7404b4284df77560f4 Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Mon, 8 Apr 2024 18:38:45 -0700 Subject: [PATCH 26/67] Update 2.8.3 Prime status --- docs/reference-guides/rancher-webhook.md | 2 +- src/pages/versions.md | 2 +- versioned_docs/version-2.8/reference-guides/rancher-webhook.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference-guides/rancher-webhook.md b/docs/reference-guides/rancher-webhook.md index c028ec47514..3b8005cba74 100644 --- a/docs/reference-guides/rancher-webhook.md +++ b/docs/reference-guides/rancher-webhook.md @@ -19,7 +19,7 @@ Each Rancher version is designed to be compatible with a single version of the w | Rancher Version | Webhook Version | Availability in Prime | Availability in Community | |-----------------|-----------------|-----------------------|---------------------------| -| v2.8.3 | v0.4.3 | ✗ | ✓ | +| v2.8.3 | v0.4.3 | ✓ | ✓ | | v2.8.2 | v0.4.2 | ✓ | ✓ | | v2.8.1 | v0.4.2 | ✓ | ✓ | | v2.8.0 | v0.4.2 | ✗ | ✓ | diff --git a/src/pages/versions.md b/src/pages/versions.md index 7eabe972ae3..7458e24a43d 100644 --- a/src/pages/versions.md +++ b/src/pages/versions.md @@ -22,7 +22,7 @@ Here you can find links to supporting documentation for the current released ver Documentation Release Notes

N/A
-
N/A
+
✓
✓
diff --git a/versioned_docs/version-2.8/reference-guides/rancher-webhook.md b/versioned_docs/version-2.8/reference-guides/rancher-webhook.md index 7aa5dfc4f1c..8fa41b14c58 100644 --- a/versioned_docs/version-2.8/reference-guides/rancher-webhook.md +++ b/versioned_docs/version-2.8/reference-guides/rancher-webhook.md @@ -20,7 +20,7 @@ Each Rancher version is designed to be compatible with a single version of the w | Rancher Version | Webhook Version | Availability in Prime | Availability in Community | |-----------------|-----------------|-----------------------|---------------------------| -| v2.8.3 | v0.4.3 | ✗ | ✓ | +| v2.8.3 | v0.4.3 | ✓ | ✓ | | v2.8.2 | v0.4.2 | ✓ | ✓ | | v2.8.1 | v0.4.2 | ✓ | ✓ | | v2.8.0 | v0.4.2 | ✗ | ✓ | From 6c24fdb4f7c2aa95c1a85a365b26196d13d5a25e Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Thu, 4 Apr 2024 16:49:43 -0700 Subject: [PATCH 27/67] Replace links to deploy-across-clusters/fleet.md --- .../installation-references/feature-flags.md | 2 +- .../resources/update-rancher-certificate.md | 4 ++-- .../enable-experimental-features/continuous-delivery.md | 2 +- .../backup-restore-and-disaster-recovery/back-up-rancher.md | 2 +- .../rancher-security/security-advisories-and-cves.md | 2 +- .../installation-references/feature-flags.md | 2 +- .../resources/update-rancher-certificate.md | 4 ++-- .../enable-experimental-features/continuous-delivery.md | 2 +- .../backup-restore-and-disaster-recovery/back-up-rancher.md | 2 +- .../rancher-security/security-advisories-and-cves.md | 2 +- .../installation-references/feature-flags.md | 2 +- .../resources/update-rancher-certificate.md | 4 ++-- .../enable-experimental-features/continuous-delivery.md | 2 +- .../backup-restore-and-disaster-recovery/back-up-rancher.md | 2 +- .../rancher-security/security-advisories-and-cves.md | 2 +- 15 files changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/getting-started/installation-and-upgrade/installation-references/feature-flags.md b/docs/getting-started/installation-and-upgrade/installation-references/feature-flags.md index f10a4fd7ec1..9661d616eba 100644 --- a/docs/getting-started/installation-and-upgrade/installation-references/feature-flags.md +++ b/docs/getting-started/installation-and-upgrade/installation-references/feature-flags.md @@ -19,7 +19,7 @@ Some feature flags require a restart of the Rancher container. Features that req The following is a list of feature flags available in Rancher. If you've upgraded from a previous Rancher version, you may see additional flags in the Rancher UI, such as `proxy` or `dashboard` (both [discontinued](/versioned_docs/version-2.5/reference-guides/installation-references/feature-flags.md)): - `continuous-delivery`: Allows Fleet GitOps to be disabled separately from Fleet. See [Continuous Delivery.](../../../how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md) for more information. -- `fleet`: The Rancher provisioning framework in v2.6 and later requires Fleet. The flag will be automatically enabled when you upgrade, even if you disabled this flag in an earlier version of Rancher. See [Fleet - GitOps at Scale](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md) for more information. +- `fleet`: The Rancher provisioning framework in v2.6 and later requires Fleet. The flag will be automatically enabled when you upgrade, even if you disabled this flag in an earlier version of Rancher. See [Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet/fleet.md) for more information. - `harvester`: Manages access to the Virtualization Management page, where users can navigate directly to Harvester clusters and access the Harvester UI. See [Harvester Integration Overview](../../../integrations-in-rancher/harvester/overview.md) for more information. - `istio-virtual-service-ui`: Enables a [visual interface](../../../how-to-guides/advanced-user-guides/enable-experimental-features/istio-traffic-management-features.md) to create, read, update, and delete Istio virtual services and destination rules, which are Istio traffic management features. - `legacy`: Enables a set of features from 2.5.x and earlier, that are slowly being phased out in favor of newer implementations. These are a mix of deprecated features as well as features that will eventually be available to newer versions. This flag is disabled by default on new Rancher installations. If you're upgrading from a previous version of Rancher, this flag is enabled. diff --git a/docs/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md b/docs/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md index 5b2e379c06b..d5eb9bfd716 100644 --- a/docs/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md +++ b/docs/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md @@ -180,7 +180,7 @@ Repeat the below steps for each downstream cluster: ### 5. Force Update Fleet clusters to reconnect the fleet-agent to Rancher -Select 'Force Update' for the clusters within the [Continuous Delivery](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. +Select 'Force Update' for the clusters within the [Continuous Delivery](../../../integrations-in-rancher/fleet/overview.md#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. #### Why is this step required? @@ -260,7 +260,7 @@ As a private CA is no longer being used, the `CATTLE_CA_CHECKSUM` environment va ### 5. Force Update Fleet clusters to reconnect the fleet-agent to Rancher -Select 'Force Update' for the clusters within the [Continuous Delivery](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. +Select 'Force Update' for the clusters within the [Continuous Delivery](../../../integrations-in-rancher/fleet/overview.md#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. #### Why is this step required? diff --git a/docs/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md b/docs/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md index 41c3aa82aea..1ea0e2eb2ea 100644 --- a/docs/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md +++ b/docs/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md @@ -6,7 +6,7 @@ title: Continuous Delivery -[Fleet](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md) comes preinstalled in Rancher can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. +[Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet/fleet.md) comes preinstalled in Rancher can't be fully disabled. comes preinstalled in Rancher can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. To enable or disable this feature, refer to the instructions on [the main page about enabling experimental features.](enable-experimental-features.md) diff --git a/docs/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md b/docs/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md index 27b02652450..9d3a6307a5f 100644 --- a/docs/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md +++ b/docs/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md @@ -40,7 +40,7 @@ Backups are created as .tar.gz files. These files can be pushed to S3 or Minio, :::note -There is a known issue in Fleet that occurs after performing a restoration using the backup-restore-operator: Secrets used for clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here](../deploy-apps-across-clusters/fleet.md#troubleshooting) for a workaround. +There is a known issue in Fleet that occurs after performing a restoration using the backup-restore-operator: Secrets used for clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here](../../../integrations-in-rancher/fleet/overview.md#troubleshooting) for a workaround. ::: diff --git a/docs/reference-guides/rancher-security/security-advisories-and-cves.md b/docs/reference-guides/rancher-security/security-advisories-and-cves.md index 0891c0d180c..c8441ae5f44 100644 --- a/docs/reference-guides/rancher-security/security-advisories-and-cves.md +++ b/docs/reference-guides/rancher-security/security-advisories-and-cves.md @@ -31,7 +31,7 @@ Rancher is committed to informing the community of security issues in our produc | [CVE-2022-21951](https://github.com/rancher/rancher/security/advisories/GHSA-vrph-m5jj-c46c) | This vulnerability only affects customers using [Weave](../../faq/container-network-interface-providers.md#weave) Container Network Interface (CNI) when configured through [RKE templates](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md). A vulnerability was discovered in Rancher versions 2.5.0 up to and including 2.5.13, and 2.6.0 up to and including 2.6.4, where a user interface (UI) issue with RKE templates does not include a value for the Weave password when Weave is chosen as the CNI. If a cluster is created based on the mentioned template, and Weave is configured as the CNI, no password will be created for [network encryption](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/) in Weave; therefore, network traffic in the cluster will be sent unencrypted. | 24 May 2022 | [Rancher v2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) and [Rancher v2.5.14](https://github.com/rancher/rancher/releases/tag/v2.5.14) | | [CVE-2021-36784](https://github.com/rancher/rancher/security/advisories/GHSA-jwvr-vv7p-gpwq) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 which allows users who have create or update permissions on [Global Roles](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md) to escalate their permissions, or those of another user, to admin-level permissions. Global Roles grant users Rancher-wide permissions, such as the ability to create clusters. In the identified versions of Rancher, when users are given permission to edit or create Global Roles, they are not restricted to only granting permissions which they already posses. This vulnerability affects customers who utilize non-admin users that are able to create or edit Global Roles. The most common use case for this scenario is the `restricted-admin` role. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | | [CVE-2021-4200](https://github.com/rancher/rancher/security/advisories/GHSA-hx8w-ghh8-r4xf) | This vulnerability only affects customers using the `restricted-admin` role in Rancher. A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 where the `global-data` role in `cattle-global-data` namespace grants write access to the Catalogs. Since each user with any level of catalog access was bound to the `global-data` role, this grants write access to templates (`CatalogTemplates`) and template versions (`CatalogTemplateVersions`) for any user with any level of catalog access. New users created in Rancher are by default assigned to the `user` role (standard user), which is not designed to grant write catalog access. This vulnerability effectively elevates the privilege of any user to write access for the catalog template and catalog template version resources. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | -| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | This vulnerability only affects customers using [Fleet](../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md) for continuous delivery with authenticated Git and/or Helm repositories. An issue was discovered in `go-getter` library in versions prior to [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) that exposes SSH private keys in base64 format due to a failure in redacting such information from error messages. The vulnerable version of this library is used in Rancher through Fleet in versions of Fleet prior to [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9). This issue affects Rancher versions 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3. The issue was found and reported by Dagan Henderson from Raft Engineering. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | +| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | This vulnerability only affects customers using [Continuous Delivery with Fleet](../../integrations-in-rancher/fleet/fleet.md) for continuous delivery with authenticated Git and/or Helm repositories. An issue was discovered in `go-getter` library in versions prior to [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) that exposes SSH private keys in base64 format due to a failure in redacting such information from error messages. The vulnerable version of this library is used in Rancher through Fleet in versions of Fleet prior to [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9). This issue affects Rancher versions 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3. The issue was found and reported by Dagan Henderson from Raft Engineering. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | | [CVE-2021-36778](https://github.com/rancher/rancher/security/advisories/GHSA-4fc7-hc63-7fjg) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.11 and from 2.6.0 up to and including 2.6.2, where an insufficient check of the same-origin policy when downloading Helm charts from a configured private repository can lead to exposure of the repository credentials to a third-party provider. This issue only happens when the user configures access credentials to a private repository in Rancher inside `Apps & Marketplace > Repositories`. The issue was found and reported by Martin Andreas Ullrich. | 14 Apr 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) and [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) | | [GHSA-hwm2-4ph6-w6m5](https://github.com/rancher/rancher/security/advisories/GHSA-hwm2-4ph6-w6m5) | A vulnerability was discovered in versions of Rancher starting 2.0 up to and including 2.6.3. The `restricted` pod security policy (PSP) provided in Rancher deviated from the upstream `restricted` policy provided in Kubernetes on account of which Rancher's PSP had `runAsUser` set to `runAsAny`, while upstream had `runAsUser` set to `MustRunAsNonRoot`. This allowed containers to run as any user, including a privileged user (`root`), even when Rancher's `restricted` policy was enforced on a project or at the cluster level. | 31 Mar 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) | | [CVE-2021-36775](https://github.com/rancher/rancher/security/advisories/GHSA-28g7-896h-695v) | A vulnerability was discovered in Rancher versions up to and including 2.4.17, 2.5.11 and 2.6.2. After removing a `Project Role` associated with a group from the project, the bindings that granted access to cluster-scoped resources for those subjects were not deleted. This was due to an incomplete authorization logic check. A user who was a member of the affected group with authenticated access to Rancher could exploit this vulnerability to access resources they shouldn't have had access to. The exposure level would depend on the original permission level granted to the affected project role. This vulnerability only affected customers using group based authentication in Rancher. | 31 Mar 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3), [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) and [Rancher v2.4.18](https://github.com/rancher/rancher/releases/tag/v2.4.18) | diff --git a/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/feature-flags.md b/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/feature-flags.md index aeba1364392..2f77729cb34 100644 --- a/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/feature-flags.md +++ b/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/feature-flags.md @@ -19,7 +19,7 @@ Some feature flags require a restart of the Rancher container. Features that req The following is a list of feature flags available in Rancher. If you've upgraded from a previous Rancher version, you may see additional flags in the Rancher UI, such as `proxy` or `dashboard` (both [discontinued](/versioned_docs/version-2.5/reference-guides/installation-references/feature-flags.md)): - `continuous-delivery`: Allows Fleet GitOps to be disabled separately from Fleet. See [Continuous Delivery.](../../../how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md) for more information. -- `fleet`: The Rancher provisioning framework in v2.6 and later requires Fleet. The flag will be automatically enabled when you upgrade, even if you disabled this flag in an earlier version of Rancher. See [Fleet - GitOps at Scale](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md) for more information. +- `fleet`: The Rancher provisioning framework in v2.6 and later requires Fleet. The flag will be automatically enabled when you upgrade, even if you disabled this flag in an earlier version of Rancher. See [Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet/fleet.md) for more information. - `harvester`: Manages access to the Virtualization Management page, where users can navigate directly to Harvester clusters and access the Harvester UI. See [Harvester Integration](../../../integrations-in-rancher/harvester.md) for more information. - `istio-virtual-service-ui`: Enables a [visual interface](../../../how-to-guides/advanced-user-guides/enable-experimental-features/istio-traffic-management-features.md) to create, read, update, and delete Istio virtual services and destination rules, which are Istio traffic management features. - `legacy`: Enables a set of features from 2.5.x and earlier, that are slowly being phased out in favor of newer implementations. These are a mix of deprecated features as well as features that will eventually be available to newer versions. This flag is disabled by default on new Rancher installations. If you're upgrading from a previous version of Rancher, this flag is enabled. diff --git a/versioned_docs/version-2.7/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md b/versioned_docs/version-2.7/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md index 5b2e379c06b..d5eb9bfd716 100644 --- a/versioned_docs/version-2.7/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md +++ b/versioned_docs/version-2.7/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md @@ -180,7 +180,7 @@ Repeat the below steps for each downstream cluster: ### 5. Force Update Fleet clusters to reconnect the fleet-agent to Rancher -Select 'Force Update' for the clusters within the [Continuous Delivery](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. +Select 'Force Update' for the clusters within the [Continuous Delivery](../../../integrations-in-rancher/fleet/overview.md#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. #### Why is this step required? @@ -260,7 +260,7 @@ As a private CA is no longer being used, the `CATTLE_CA_CHECKSUM` environment va ### 5. Force Update Fleet clusters to reconnect the fleet-agent to Rancher -Select 'Force Update' for the clusters within the [Continuous Delivery](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. +Select 'Force Update' for the clusters within the [Continuous Delivery](../../../integrations-in-rancher/fleet/overview.md#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. #### Why is this step required? diff --git a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md index 41c3aa82aea..1ea0e2eb2ea 100644 --- a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md +++ b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md @@ -6,7 +6,7 @@ title: Continuous Delivery -[Fleet](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md) comes preinstalled in Rancher can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. +[Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet/fleet.md) comes preinstalled in Rancher can't be fully disabled. comes preinstalled in Rancher can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. To enable or disable this feature, refer to the instructions on [the main page about enabling experimental features.](enable-experimental-features.md) diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md index 27b02652450..9d3a6307a5f 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md @@ -40,7 +40,7 @@ Backups are created as .tar.gz files. These files can be pushed to S3 or Minio, :::note -There is a known issue in Fleet that occurs after performing a restoration using the backup-restore-operator: Secrets used for clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here](../deploy-apps-across-clusters/fleet.md#troubleshooting) for a workaround. +There is a known issue in Fleet that occurs after performing a restoration using the backup-restore-operator: Secrets used for clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here](../../../integrations-in-rancher/fleet/overview.md#troubleshooting) for a workaround. ::: diff --git a/versioned_docs/version-2.7/reference-guides/rancher-security/security-advisories-and-cves.md b/versioned_docs/version-2.7/reference-guides/rancher-security/security-advisories-and-cves.md index a142092fa2d..c3fd24188aa 100644 --- a/versioned_docs/version-2.7/reference-guides/rancher-security/security-advisories-and-cves.md +++ b/versioned_docs/version-2.7/reference-guides/rancher-security/security-advisories-and-cves.md @@ -30,7 +30,7 @@ Rancher is committed to informing the community of security issues in our produc | [CVE-2022-21951](https://github.com/rancher/rancher/security/advisories/GHSA-vrph-m5jj-c46c) | This vulnerability only affects customers using [Weave](../../faq/container-network-interface-providers.md#weave) Container Network Interface (CNI) when configured through [RKE templates](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md). A vulnerability was discovered in Rancher versions 2.5.0 up to and including 2.5.13, and 2.6.0 up to and including 2.6.4, where a user interface (UI) issue with RKE templates does not include a value for the Weave password when Weave is chosen as the CNI. If a cluster is created based on the mentioned template, and Weave is configured as the CNI, no password will be created for [network encryption](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/) in Weave; therefore, network traffic in the cluster will be sent unencrypted. | 24 May 2022 | [Rancher v2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) and [Rancher v2.5.14](https://github.com/rancher/rancher/releases/tag/v2.5.14) | | [CVE-2021-36784](https://github.com/rancher/rancher/security/advisories/GHSA-jwvr-vv7p-gpwq) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 which allows users who have create or update permissions on [Global Roles](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md) to escalate their permissions, or those of another user, to admin-level permissions. Global Roles grant users Rancher-wide permissions, such as the ability to create clusters. In the identified versions of Rancher, when users are given permission to edit or create Global Roles, they are not restricted to only granting permissions which they already posses. This vulnerability affects customers who utilize non-admin users that are able to create or edit Global Roles. The most common use case for this scenario is the `restricted-admin` role. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | | [CVE-2021-4200](https://github.com/rancher/rancher/security/advisories/GHSA-hx8w-ghh8-r4xf) | This vulnerability only affects customers using the `restricted-admin` role in Rancher. A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 where the `global-data` role in `cattle-global-data` namespace grants write access to the Catalogs. Since each user with any level of catalog access was bound to the `global-data` role, this grants write access to templates (`CatalogTemplates`) and template versions (`CatalogTemplateVersions`) for any user with any level of catalog access. New users created in Rancher are by default assigned to the `user` role (standard user), which is not designed to grant write catalog access. This vulnerability effectively elevates the privilege of any user to write access for the catalog template and catalog template version resources. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | -| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | This vulnerability only affects customers using [Fleet](../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md) for continuous delivery with authenticated Git and/or Helm repositories. An issue was discovered in `go-getter` library in versions prior to [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) that exposes SSH private keys in base64 format due to a failure in redacting such information from error messages. The vulnerable version of this library is used in Rancher through Fleet in versions of Fleet prior to [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9). This issue affects Rancher versions 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3. The issue was found and reported by Dagan Henderson from Raft Engineering. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | +| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | This vulnerability only affects customers using [Continuous Delivery with Fleet](../../integrations-in-rancher/fleet/fleet.md) for continuous delivery with authenticated Git and/or Helm repositories. An issue was discovered in `go-getter` library in versions prior to [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) that exposes SSH private keys in base64 format due to a failure in redacting such information from error messages. The vulnerable version of this library is used in Rancher through Fleet in versions of Fleet prior to [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9). This issue affects Rancher versions 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3. The issue was found and reported by Dagan Henderson from Raft Engineering. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | | [CVE-2021-36778](https://github.com/rancher/rancher/security/advisories/GHSA-4fc7-hc63-7fjg) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.11 and from 2.6.0 up to and including 2.6.2, where an insufficient check of the same-origin policy when downloading Helm charts from a configured private repository can lead to exposure of the repository credentials to a third-party provider. This issue only happens when the user configures access credentials to a private repository in Rancher inside `Apps & Marketplace > Repositories`. The issue was found and reported by Martin Andreas Ullrich. | 14 Apr 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) and [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) | | [GHSA-hwm2-4ph6-w6m5](https://github.com/rancher/rancher/security/advisories/GHSA-hwm2-4ph6-w6m5) | A vulnerability was discovered in versions of Rancher starting 2.0 up to and including 2.6.3. The `restricted` pod security policy (PSP) provided in Rancher deviated from the upstream `restricted` policy provided in Kubernetes on account of which Rancher's PSP had `runAsUser` set to `runAsAny`, while upstream had `runAsUser` set to `MustRunAsNonRoot`. This allowed containers to run as any user, including a privileged user (`root`), even when Rancher's `restricted` policy was enforced on a project or at the cluster level. | 31 Mar 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) | | [CVE-2021-36775](https://github.com/rancher/rancher/security/advisories/GHSA-28g7-896h-695v) | A vulnerability was discovered in Rancher versions up to and including 2.4.17, 2.5.11 and 2.6.2. After removing a `Project Role` associated with a group from the project, the bindings that granted access to cluster-scoped resources for those subjects were not deleted. This was due to an incomplete authorization logic check. A user who was a member of the affected group with authenticated access to Rancher could exploit this vulnerability to access resources they shouldn't have had access to. The exposure level would depend on the original permission level granted to the affected project role. This vulnerability only affected customers using group based authentication in Rancher. | 31 Mar 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3), [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) and [Rancher v2.4.18](https://github.com/rancher/rancher/releases/tag/v2.4.18) | diff --git a/versioned_docs/version-2.8/getting-started/installation-and-upgrade/installation-references/feature-flags.md b/versioned_docs/version-2.8/getting-started/installation-and-upgrade/installation-references/feature-flags.md index f10a4fd7ec1..9661d616eba 100644 --- a/versioned_docs/version-2.8/getting-started/installation-and-upgrade/installation-references/feature-flags.md +++ b/versioned_docs/version-2.8/getting-started/installation-and-upgrade/installation-references/feature-flags.md @@ -19,7 +19,7 @@ Some feature flags require a restart of the Rancher container. Features that req The following is a list of feature flags available in Rancher. If you've upgraded from a previous Rancher version, you may see additional flags in the Rancher UI, such as `proxy` or `dashboard` (both [discontinued](/versioned_docs/version-2.5/reference-guides/installation-references/feature-flags.md)): - `continuous-delivery`: Allows Fleet GitOps to be disabled separately from Fleet. See [Continuous Delivery.](../../../how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md) for more information. -- `fleet`: The Rancher provisioning framework in v2.6 and later requires Fleet. The flag will be automatically enabled when you upgrade, even if you disabled this flag in an earlier version of Rancher. See [Fleet - GitOps at Scale](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md) for more information. +- `fleet`: The Rancher provisioning framework in v2.6 and later requires Fleet. The flag will be automatically enabled when you upgrade, even if you disabled this flag in an earlier version of Rancher. See [Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet/fleet.md) for more information. - `harvester`: Manages access to the Virtualization Management page, where users can navigate directly to Harvester clusters and access the Harvester UI. See [Harvester Integration Overview](../../../integrations-in-rancher/harvester/overview.md) for more information. - `istio-virtual-service-ui`: Enables a [visual interface](../../../how-to-guides/advanced-user-guides/enable-experimental-features/istio-traffic-management-features.md) to create, read, update, and delete Istio virtual services and destination rules, which are Istio traffic management features. - `legacy`: Enables a set of features from 2.5.x and earlier, that are slowly being phased out in favor of newer implementations. These are a mix of deprecated features as well as features that will eventually be available to newer versions. This flag is disabled by default on new Rancher installations. If you're upgrading from a previous version of Rancher, this flag is enabled. diff --git a/versioned_docs/version-2.8/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md b/versioned_docs/version-2.8/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md index 5b2e379c06b..d5eb9bfd716 100644 --- a/versioned_docs/version-2.8/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md +++ b/versioned_docs/version-2.8/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md @@ -180,7 +180,7 @@ Repeat the below steps for each downstream cluster: ### 5. Force Update Fleet clusters to reconnect the fleet-agent to Rancher -Select 'Force Update' for the clusters within the [Continuous Delivery](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. +Select 'Force Update' for the clusters within the [Continuous Delivery](../../../integrations-in-rancher/fleet/overview.md#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. #### Why is this step required? @@ -260,7 +260,7 @@ As a private CA is no longer being used, the `CATTLE_CA_CHECKSUM` environment va ### 5. Force Update Fleet clusters to reconnect the fleet-agent to Rancher -Select 'Force Update' for the clusters within the [Continuous Delivery](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. +Select 'Force Update' for the clusters within the [Continuous Delivery](../../../integrations-in-rancher/fleet/overview.md#accessing-fleet-in-the-rancher-ui) view of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher. #### Why is this step required? diff --git a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md index 41c3aa82aea..1ea0e2eb2ea 100644 --- a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md +++ b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md @@ -6,7 +6,7 @@ title: Continuous Delivery -[Fleet](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md) comes preinstalled in Rancher can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. +[Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet/fleet.md) comes preinstalled in Rancher can't be fully disabled. comes preinstalled in Rancher can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. To enable or disable this feature, refer to the instructions on [the main page about enabling experimental features.](enable-experimental-features.md) diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md index 27b02652450..9d3a6307a5f 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md @@ -40,7 +40,7 @@ Backups are created as .tar.gz files. These files can be pushed to S3 or Minio, :::note -There is a known issue in Fleet that occurs after performing a restoration using the backup-restore-operator: Secrets used for clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here](../deploy-apps-across-clusters/fleet.md#troubleshooting) for a workaround. +There is a known issue in Fleet that occurs after performing a restoration using the backup-restore-operator: Secrets used for clientSecretName and helmSecretName are not included in Fleet gitrepos. Refer [here](../../../integrations-in-rancher/fleet/overview.md#troubleshooting) for a workaround. ::: diff --git a/versioned_docs/version-2.8/reference-guides/rancher-security/security-advisories-and-cves.md b/versioned_docs/version-2.8/reference-guides/rancher-security/security-advisories-and-cves.md index a142092fa2d..c3fd24188aa 100644 --- a/versioned_docs/version-2.8/reference-guides/rancher-security/security-advisories-and-cves.md +++ b/versioned_docs/version-2.8/reference-guides/rancher-security/security-advisories-and-cves.md @@ -30,7 +30,7 @@ Rancher is committed to informing the community of security issues in our produc | [CVE-2022-21951](https://github.com/rancher/rancher/security/advisories/GHSA-vrph-m5jj-c46c) | This vulnerability only affects customers using [Weave](../../faq/container-network-interface-providers.md#weave) Container Network Interface (CNI) when configured through [RKE templates](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md). A vulnerability was discovered in Rancher versions 2.5.0 up to and including 2.5.13, and 2.6.0 up to and including 2.6.4, where a user interface (UI) issue with RKE templates does not include a value for the Weave password when Weave is chosen as the CNI. If a cluster is created based on the mentioned template, and Weave is configured as the CNI, no password will be created for [network encryption](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/) in Weave; therefore, network traffic in the cluster will be sent unencrypted. | 24 May 2022 | [Rancher v2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) and [Rancher v2.5.14](https://github.com/rancher/rancher/releases/tag/v2.5.14) | | [CVE-2021-36784](https://github.com/rancher/rancher/security/advisories/GHSA-jwvr-vv7p-gpwq) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 which allows users who have create or update permissions on [Global Roles](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md) to escalate their permissions, or those of another user, to admin-level permissions. Global Roles grant users Rancher-wide permissions, such as the ability to create clusters. In the identified versions of Rancher, when users are given permission to edit or create Global Roles, they are not restricted to only granting permissions which they already posses. This vulnerability affects customers who utilize non-admin users that are able to create or edit Global Roles. The most common use case for this scenario is the `restricted-admin` role. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | | [CVE-2021-4200](https://github.com/rancher/rancher/security/advisories/GHSA-hx8w-ghh8-r4xf) | This vulnerability only affects customers using the `restricted-admin` role in Rancher. A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 where the `global-data` role in `cattle-global-data` namespace grants write access to the Catalogs. Since each user with any level of catalog access was bound to the `global-data` role, this grants write access to templates (`CatalogTemplates`) and template versions (`CatalogTemplateVersions`) for any user with any level of catalog access. New users created in Rancher are by default assigned to the `user` role (standard user), which is not designed to grant write catalog access. This vulnerability effectively elevates the privilege of any user to write access for the catalog template and catalog template version resources. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | -| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | This vulnerability only affects customers using [Fleet](../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md) for continuous delivery with authenticated Git and/or Helm repositories. An issue was discovered in `go-getter` library in versions prior to [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) that exposes SSH private keys in base64 format due to a failure in redacting such information from error messages. The vulnerable version of this library is used in Rancher through Fleet in versions of Fleet prior to [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9). This issue affects Rancher versions 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3. The issue was found and reported by Dagan Henderson from Raft Engineering. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | +| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | This vulnerability only affects customers using [Continuous Delivery with Fleet](../../integrations-in-rancher/fleet/fleet.md) for continuous delivery with authenticated Git and/or Helm repositories. An issue was discovered in `go-getter` library in versions prior to [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) that exposes SSH private keys in base64 format due to a failure in redacting such information from error messages. The vulnerable version of this library is used in Rancher through Fleet in versions of Fleet prior to [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9). This issue affects Rancher versions 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3. The issue was found and reported by Dagan Henderson from Raft Engineering. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | | [CVE-2021-36778](https://github.com/rancher/rancher/security/advisories/GHSA-4fc7-hc63-7fjg) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.11 and from 2.6.0 up to and including 2.6.2, where an insufficient check of the same-origin policy when downloading Helm charts from a configured private repository can lead to exposure of the repository credentials to a third-party provider. This issue only happens when the user configures access credentials to a private repository in Rancher inside `Apps & Marketplace > Repositories`. The issue was found and reported by Martin Andreas Ullrich. | 14 Apr 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) and [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) | | [GHSA-hwm2-4ph6-w6m5](https://github.com/rancher/rancher/security/advisories/GHSA-hwm2-4ph6-w6m5) | A vulnerability was discovered in versions of Rancher starting 2.0 up to and including 2.6.3. The `restricted` pod security policy (PSP) provided in Rancher deviated from the upstream `restricted` policy provided in Kubernetes on account of which Rancher's PSP had `runAsUser` set to `runAsAny`, while upstream had `runAsUser` set to `MustRunAsNonRoot`. This allowed containers to run as any user, including a privileged user (`root`), even when Rancher's `restricted` policy was enforced on a project or at the cluster level. | 31 Mar 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) | | [CVE-2021-36775](https://github.com/rancher/rancher/security/advisories/GHSA-28g7-896h-695v) | A vulnerability was discovered in Rancher versions up to and including 2.4.17, 2.5.11 and 2.6.2. After removing a `Project Role` associated with a group from the project, the bindings that granted access to cluster-scoped resources for those subjects were not deleted. This was due to an incomplete authorization logic check. A user who was a member of the affected group with authenticated access to Rancher could exploit this vulnerability to access resources they shouldn't have had access to. The exposure level would depend on the original permission level granted to the affected project role. This vulnerability only affected customers using group based authentication in Rancher. | 31 Mar 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3), [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) and [Rancher v2.4.18](https://github.com/rancher/rancher/releases/tag/v2.4.18) | From 053a52900a79b9654e196b278ec3f1e49bb61a9d Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Mon, 8 Apr 2024 17:01:40 -0700 Subject: [PATCH 28/67] Remove references to multi-cluster apps feature --- .../about-rke1-templates/infrastructure.md | 1 - .../authentication-config/authentication-config.md | 2 +- .../authentication-config/manage-users-and-groups.md | 2 +- docs/integrations-in-rancher/harvester/overview.md | 2 +- .../about-rke1-templates/infrastructure.md | 1 - .../authentication-config/authentication-config.md | 2 +- .../authentication-config/manage-users-and-groups.md | 2 +- versioned_docs/version-2.7/integrations-in-rancher/harvester.md | 2 +- .../about-rke1-templates/infrastructure.md | 1 - .../authentication-config/authentication-config.md | 2 +- .../authentication-config/manage-users-and-groups.md | 2 +- .../version-2.8/integrations-in-rancher/harvester/overview.md | 2 +- 12 files changed, 9 insertions(+), 12 deletions(-) diff --git a/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/infrastructure.md b/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/infrastructure.md index ae3be31fa37..65ccff89e06 100644 --- a/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/infrastructure.md +++ b/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/infrastructure.md @@ -25,7 +25,6 @@ This section focuses on how to use Terraform with the [Rancher 2 Terraform provi Terraform allows you to: - Define almost any kind of infrastructure-as-code, including servers, databases, load balancers, monitoring, firewall settings, and SSL certificates -- Leverage catalog apps and multi-cluster apps - Codify infrastructure across many platforms, including Rancher and major cloud providers - Commit infrastructure-as-code to version control - Easily repeat configuration and setup of infrastructure diff --git a/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md b/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md index a05fc7c93a3..b923d862b5c 100644 --- a/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md +++ b/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md @@ -42,7 +42,7 @@ In most cases, you should use an external authentication service over local auth ## Users and Groups -Rancher relies on users and groups to determine who is allowed to log in to Rancher and which resources they can access. When authenticating with an external provider, groups are provided from the external provider based on the user. These users and groups are given specific roles to resources like clusters, projects, multi-cluster apps, and global DNS providers and entries. When you give access to a group, all users who are a member of that group in the authentication provider will be able to access the resource with the permissions that you've specified. For more information on roles and permissions, see [Role Based Access Control](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md). +Rancher relies on users and groups to determine who is allowed to log in to Rancher and which resources they can access. When authenticating with an external provider, groups are provided from the external provider based on the user. These users and groups are given specific roles to resources like clusters, projects, and global DNS providers and entries. When you give access to a group, all users who are a member of that group in the authentication provider will be able to access the resource with the permissions that you've specified. For more information on roles and permissions, see [Role Based Access Control](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md). :::note diff --git a/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md b/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md index ec39be1f01b..8b66317aabb 100644 --- a/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md +++ b/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md @@ -8,7 +8,7 @@ title: Users and Groups Rancher relies on users and groups to determine who is allowed to log in to Rancher and which resources they can access. When you configure an external authentication provider, users from that provider will be able to log in to your Rancher server. When a user logs in, the authentication provider will supply your Rancher server with a list of groups to which the user belongs. -Access to clusters, projects, multi-cluster apps, and global DNS providers and entries can be controlled by adding either individual users or groups to these resources. When you add a group to a resource, all users who are members of that group in the authentication provider, will be able to access the resource with the permissions that you've specified for the group. For more information on roles and permissions, see [Role Based Access Control](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md). +Access to clusters, projects, and global DNS providers and entries can be controlled by adding either individual users or groups to these resources. When you add a group to a resource, all users who are members of that group in the authentication provider, will be able to access the resource with the permissions that you've specified for the group. For more information on roles and permissions, see [Role Based Access Control](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md). ## Managing Members diff --git a/docs/integrations-in-rancher/harvester/overview.md b/docs/integrations-in-rancher/harvester/overview.md index cbf94de9cd3..d22afe15965 100644 --- a/docs/integrations-in-rancher/harvester/overview.md +++ b/docs/integrations-in-rancher/harvester/overview.md @@ -14,7 +14,7 @@ The Harvester feature flag is used to manage access to the Virtualization Manage To navigate to the Harvester cluster, click **☰ > Virtualization Management**. From Harvester Clusters page, click one of the clusters listed to go to the single Harvester cluster view. -* If the Harvester feature flag is enabled, Harvester clusters will be filtered out from any pages or apps (such as Fleet and the multi-cluster app) that list Kubernetes clusters. +* If the Harvester feature flag is enabled, Harvester clusters will be filtered out from any pages or apps (such as Continuous Delivery with Fleet) that list Kubernetes clusters. * If the Harvester feature flag is disabled, and a Harvester cluster is imported, the Harvester cluster will be shown in the Rancher cluster list in the Cluster Management page. Harvester clusters will only be shown on the cluster list when the feature flag is off. diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/infrastructure.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/infrastructure.md index ae3be31fa37..65ccff89e06 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/infrastructure.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/infrastructure.md @@ -25,7 +25,6 @@ This section focuses on how to use Terraform with the [Rancher 2 Terraform provi Terraform allows you to: - Define almost any kind of infrastructure-as-code, including servers, databases, load balancers, monitoring, firewall settings, and SSL certificates -- Leverage catalog apps and multi-cluster apps - Codify infrastructure across many platforms, including Rancher and major cloud providers - Commit infrastructure-as-code to version control - Easily repeat configuration and setup of infrastructure diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md index a05fc7c93a3..b923d862b5c 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md @@ -42,7 +42,7 @@ In most cases, you should use an external authentication service over local auth ## Users and Groups -Rancher relies on users and groups to determine who is allowed to log in to Rancher and which resources they can access. When authenticating with an external provider, groups are provided from the external provider based on the user. These users and groups are given specific roles to resources like clusters, projects, multi-cluster apps, and global DNS providers and entries. When you give access to a group, all users who are a member of that group in the authentication provider will be able to access the resource with the permissions that you've specified. For more information on roles and permissions, see [Role Based Access Control](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md). +Rancher relies on users and groups to determine who is allowed to log in to Rancher and which resources they can access. When authenticating with an external provider, groups are provided from the external provider based on the user. These users and groups are given specific roles to resources like clusters, projects, and global DNS providers and entries. When you give access to a group, all users who are a member of that group in the authentication provider will be able to access the resource with the permissions that you've specified. For more information on roles and permissions, see [Role Based Access Control](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md). :::note diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md index ec39be1f01b..8b66317aabb 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md @@ -8,7 +8,7 @@ title: Users and Groups Rancher relies on users and groups to determine who is allowed to log in to Rancher and which resources they can access. When you configure an external authentication provider, users from that provider will be able to log in to your Rancher server. When a user logs in, the authentication provider will supply your Rancher server with a list of groups to which the user belongs. -Access to clusters, projects, multi-cluster apps, and global DNS providers and entries can be controlled by adding either individual users or groups to these resources. When you add a group to a resource, all users who are members of that group in the authentication provider, will be able to access the resource with the permissions that you've specified for the group. For more information on roles and permissions, see [Role Based Access Control](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md). +Access to clusters, projects, and global DNS providers and entries can be controlled by adding either individual users or groups to these resources. When you add a group to a resource, all users who are members of that group in the authentication provider, will be able to access the resource with the permissions that you've specified for the group. For more information on roles and permissions, see [Role Based Access Control](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md). ## Managing Members diff --git a/versioned_docs/version-2.7/integrations-in-rancher/harvester.md b/versioned_docs/version-2.7/integrations-in-rancher/harvester.md index cd6673582d8..1efef3c7547 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/harvester.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/harvester.md @@ -14,7 +14,7 @@ The Harvester feature flag is used to manage access to the Virtualization Manage To navigate to the Harvester cluster, click **☰ > Virtualization Management**. From Harvester Clusters page, click one of the clusters listed to go to the single Harvester cluster view. -* If the Harvester feature flag is enabled, Harvester clusters will be filtered out from any pages or apps (such as Fleet and the multi-cluster app) that list Kubernetes clusters. +* If the Harvester feature flag is enabled, Harvester clusters will be filtered out from any pages or apps (such as Continuous Delivery with Fleet) that list Kubernetes clusters. * If the Harvester feature flag is disabled, and a Harvester cluster is imported, the Harvester cluster will be shown in the Rancher cluster list in the Cluster Management page. Harvester clusters will only be shown on the cluster list when the feature flag is off. diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/infrastructure.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/infrastructure.md index ae3be31fa37..65ccff89e06 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/infrastructure.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/infrastructure.md @@ -25,7 +25,6 @@ This section focuses on how to use Terraform with the [Rancher 2 Terraform provi Terraform allows you to: - Define almost any kind of infrastructure-as-code, including servers, databases, load balancers, monitoring, firewall settings, and SSL certificates -- Leverage catalog apps and multi-cluster apps - Codify infrastructure across many platforms, including Rancher and major cloud providers - Commit infrastructure-as-code to version control - Easily repeat configuration and setup of infrastructure diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md index 23bc98fbdb3..7b7128852be 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md @@ -42,7 +42,7 @@ In most cases, you should use an external authentication service over local auth ## Users and Groups -Rancher relies on users and groups to determine who is allowed to log in to Rancher and which resources they can access. When authenticating with an external provider, groups are provided from the external provider based on the user. These users and groups are given specific roles to resources like clusters, projects, multi-cluster apps, and global DNS providers and entries. When you give access to a group, all users who are a member of that group in the authentication provider will be able to access the resource with the permissions that you've specified. For more information on roles and permissions, see [Role Based Access Control](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md). +Rancher relies on users and groups to determine who is allowed to log in to Rancher and which resources they can access. When authenticating with an external provider, groups are provided from the external provider based on the user. These users and groups are given specific roles to resources like clusters, projects, and global DNS providers and entries. When you give access to a group, all users who are a member of that group in the authentication provider will be able to access the resource with the permissions that you've specified. For more information on roles and permissions, see [Role Based Access Control](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md). :::note diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md index ec39be1f01b..8b66317aabb 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md @@ -8,7 +8,7 @@ title: Users and Groups Rancher relies on users and groups to determine who is allowed to log in to Rancher and which resources they can access. When you configure an external authentication provider, users from that provider will be able to log in to your Rancher server. When a user logs in, the authentication provider will supply your Rancher server with a list of groups to which the user belongs. -Access to clusters, projects, multi-cluster apps, and global DNS providers and entries can be controlled by adding either individual users or groups to these resources. When you add a group to a resource, all users who are members of that group in the authentication provider, will be able to access the resource with the permissions that you've specified for the group. For more information on roles and permissions, see [Role Based Access Control](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md). +Access to clusters, projects, and global DNS providers and entries can be controlled by adding either individual users or groups to these resources. When you add a group to a resource, all users who are members of that group in the authentication provider, will be able to access the resource with the permissions that you've specified for the group. For more information on roles and permissions, see [Role Based Access Control](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md). ## Managing Members diff --git a/versioned_docs/version-2.8/integrations-in-rancher/harvester/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/harvester/overview.md index cbf94de9cd3..d22afe15965 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/harvester/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/harvester/overview.md @@ -14,7 +14,7 @@ The Harvester feature flag is used to manage access to the Virtualization Manage To navigate to the Harvester cluster, click **☰ > Virtualization Management**. From Harvester Clusters page, click one of the clusters listed to go to the single Harvester cluster view. -* If the Harvester feature flag is enabled, Harvester clusters will be filtered out from any pages or apps (such as Fleet and the multi-cluster app) that list Kubernetes clusters. +* If the Harvester feature flag is enabled, Harvester clusters will be filtered out from any pages or apps (such as Continuous Delivery with Fleet) that list Kubernetes clusters. * If the Harvester feature flag is disabled, and a Harvester cluster is imported, the Harvester cluster will be shown in the Rancher cluster list in the Cluster Management page. Harvester clusters will only be shown on the cluster list when the feature flag is off. From 016670ac2185ac79cb3a5b4fcb97d796f7a5da8c Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Mon, 8 Apr 2024 18:26:29 -0700 Subject: [PATCH 29/67] Update sidebars --- sidebars.js | 12 ------------ versioned_sidebars/version-2.7-sidebars.json | 12 ------------ versioned_sidebars/version-2.8-sidebars.json | 12 ------------ 3 files changed, 36 deletions(-) diff --git a/sidebars.js b/sidebars.js index 1ee600f77d4..7e25ab4b5fe 100644 --- a/sidebars.js +++ b/sidebars.js @@ -634,18 +634,6 @@ const sidebars = { "how-to-guides/new-user-guides/helm-charts-in-rancher/create-apps", ] }, - { - type: 'category', - label: 'Deploying Apps Across Clusters', - link: { - type: 'doc', - id: "how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters", - }, - items: [ - "how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet", - "how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps", - ] - }, { type: 'category', label: 'Backup, Restore, and Disaster Recovery', diff --git a/versioned_sidebars/version-2.7-sidebars.json b/versioned_sidebars/version-2.7-sidebars.json index 7d677be558d..378b51be530 100644 --- a/versioned_sidebars/version-2.7-sidebars.json +++ b/versioned_sidebars/version-2.7-sidebars.json @@ -598,18 +598,6 @@ "how-to-guides/new-user-guides/helm-charts-in-rancher/create-apps" ] }, - { - "type": "category", - "label": "Deploying Apps Across Clusters", - "link": { - "type": "doc", - "id": "how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters" - }, - "items": [ - "how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet", - "how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps" - ] - }, { "type": "category", "label": "Backup, Restore, and Disaster Recovery", diff --git a/versioned_sidebars/version-2.8-sidebars.json b/versioned_sidebars/version-2.8-sidebars.json index 3e770dd537d..388856758be 100644 --- a/versioned_sidebars/version-2.8-sidebars.json +++ b/versioned_sidebars/version-2.8-sidebars.json @@ -598,18 +598,6 @@ "how-to-guides/new-user-guides/helm-charts-in-rancher/create-apps" ] }, - { - "type": "category", - "label": "Deploying Apps Across Clusters", - "link": { - "type": "doc", - "id": "how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters" - }, - "items": [ - "how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet", - "how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps" - ] - }, { "type": "category", "label": "Backup, Restore, and Disaster Recovery", From f2711e722fb04faabc676bab44a32ea4067dec20 Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Mon, 8 Apr 2024 18:32:27 -0700 Subject: [PATCH 30/67] Add redirects --- docusaurus.config.js | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docusaurus.config.js b/docusaurus.config.js index b75e5b9676f..7c886899f1b 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -245,6 +245,18 @@ module.exports = { { fromExtensions: ['html', 'htm'], redirects: [ + { // Redirects for multi-cluster apps removal (rancher-docs/issues/734) + to: '/integrations-in-rancher/fleet', + from: ['/how-to-guides/new-user-guides/deploy-apps-across-clusters','/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet','/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps'] + }, + { + to: '/v2.8/integrations-in-rancher/fleet', + from: ['/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters','/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet','/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps'] + }, + { + to: '/v2.7/integrations-in-rancher/fleet', + from: ['/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters','/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet','/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps'] + },// Redirects for multi-cluster apps removal (rancher-docs/issues/734) (end) { // Redirects for pages-for-subheaders removal [2.0-2.4] to: '/v2.0-v2.4/how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-authentication', from: '/v2.0-v2.4/pages-for-subheaders/about-authentication' From 2fa977d4a398c41d77b1ed12dfe43e8400a2ad5e Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Mon, 8 Apr 2024 19:03:36 -0700 Subject: [PATCH 31/67] Fix Fleet integration page/URL --- docusaurus.config.js | 2 +- .../installation-references/feature-flags.md | 2 +- .../enable-experimental-features/continuous-delivery.md | 2 +- .../rancher-security/security-advisories-and-cves.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docusaurus.config.js b/docusaurus.config.js index 0a9bf6e6ef7..e39fa77d9fe 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -254,7 +254,7 @@ module.exports = { from: ['/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters','/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet','/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps'] }, { - to: '/v2.7/integrations-in-rancher/fleet', + to: '/v2.7/integrations-in-rancher/fleet-gitops-at-scale', from: ['/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters','/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet','/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps'] },// Redirects for multi-cluster apps removal (rancher-docs/issues/734) (end) { diff --git a/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/feature-flags.md b/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/feature-flags.md index 2f77729cb34..ee9f47347dd 100644 --- a/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/feature-flags.md +++ b/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/feature-flags.md @@ -19,7 +19,7 @@ Some feature flags require a restart of the Rancher container. Features that req The following is a list of feature flags available in Rancher. If you've upgraded from a previous Rancher version, you may see additional flags in the Rancher UI, such as `proxy` or `dashboard` (both [discontinued](/versioned_docs/version-2.5/reference-guides/installation-references/feature-flags.md)): - `continuous-delivery`: Allows Fleet GitOps to be disabled separately from Fleet. See [Continuous Delivery.](../../../how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md) for more information. -- `fleet`: The Rancher provisioning framework in v2.6 and later requires Fleet. The flag will be automatically enabled when you upgrade, even if you disabled this flag in an earlier version of Rancher. See [Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet/fleet.md) for more information. +- `fleet`: The Rancher provisioning framework in v2.6 and later requires Fleet. The flag will be automatically enabled when you upgrade, even if you disabled this flag in an earlier version of Rancher. See [Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet-gitops-at-scale/fleet-gitops-at-scale.md) for more information. - `harvester`: Manages access to the Virtualization Management page, where users can navigate directly to Harvester clusters and access the Harvester UI. See [Harvester Integration](../../../integrations-in-rancher/harvester.md) for more information. - `istio-virtual-service-ui`: Enables a [visual interface](../../../how-to-guides/advanced-user-guides/enable-experimental-features/istio-traffic-management-features.md) to create, read, update, and delete Istio virtual services and destination rules, which are Istio traffic management features. - `legacy`: Enables a set of features from 2.5.x and earlier, that are slowly being phased out in favor of newer implementations. These are a mix of deprecated features as well as features that will eventually be available to newer versions. This flag is disabled by default on new Rancher installations. If you're upgrading from a previous version of Rancher, this flag is enabled. diff --git a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md index 1ea0e2eb2ea..8446287f45f 100644 --- a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md +++ b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md @@ -6,7 +6,7 @@ title: Continuous Delivery -[Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet/fleet.md) comes preinstalled in Rancher can't be fully disabled. comes preinstalled in Rancher can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. +[Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet-gitops-at-scale/fleet-gitops-at-scale.md) comes preinstalled in Rancher can't be fully disabled. comes preinstalled in Rancher can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. To enable or disable this feature, refer to the instructions on [the main page about enabling experimental features.](enable-experimental-features.md) diff --git a/versioned_docs/version-2.7/reference-guides/rancher-security/security-advisories-and-cves.md b/versioned_docs/version-2.7/reference-guides/rancher-security/security-advisories-and-cves.md index c3fd24188aa..3ed161f7e08 100644 --- a/versioned_docs/version-2.7/reference-guides/rancher-security/security-advisories-and-cves.md +++ b/versioned_docs/version-2.7/reference-guides/rancher-security/security-advisories-and-cves.md @@ -30,7 +30,7 @@ Rancher is committed to informing the community of security issues in our produc | [CVE-2022-21951](https://github.com/rancher/rancher/security/advisories/GHSA-vrph-m5jj-c46c) | This vulnerability only affects customers using [Weave](../../faq/container-network-interface-providers.md#weave) Container Network Interface (CNI) when configured through [RKE templates](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md). A vulnerability was discovered in Rancher versions 2.5.0 up to and including 2.5.13, and 2.6.0 up to and including 2.6.4, where a user interface (UI) issue with RKE templates does not include a value for the Weave password when Weave is chosen as the CNI. If a cluster is created based on the mentioned template, and Weave is configured as the CNI, no password will be created for [network encryption](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/) in Weave; therefore, network traffic in the cluster will be sent unencrypted. | 24 May 2022 | [Rancher v2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) and [Rancher v2.5.14](https://github.com/rancher/rancher/releases/tag/v2.5.14) | | [CVE-2021-36784](https://github.com/rancher/rancher/security/advisories/GHSA-jwvr-vv7p-gpwq) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 which allows users who have create or update permissions on [Global Roles](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md) to escalate their permissions, or those of another user, to admin-level permissions. Global Roles grant users Rancher-wide permissions, such as the ability to create clusters. In the identified versions of Rancher, when users are given permission to edit or create Global Roles, they are not restricted to only granting permissions which they already posses. This vulnerability affects customers who utilize non-admin users that are able to create or edit Global Roles. The most common use case for this scenario is the `restricted-admin` role. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | | [CVE-2021-4200](https://github.com/rancher/rancher/security/advisories/GHSA-hx8w-ghh8-r4xf) | This vulnerability only affects customers using the `restricted-admin` role in Rancher. A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3 where the `global-data` role in `cattle-global-data` namespace grants write access to the Catalogs. Since each user with any level of catalog access was bound to the `global-data` role, this grants write access to templates (`CatalogTemplates`) and template versions (`CatalogTemplateVersions`) for any user with any level of catalog access. New users created in Rancher are by default assigned to the `user` role (standard user), which is not designed to grant write catalog access. This vulnerability effectively elevates the privilege of any user to write access for the catalog template and catalog template version resources. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | -| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | This vulnerability only affects customers using [Continuous Delivery with Fleet](../../integrations-in-rancher/fleet/fleet.md) for continuous delivery with authenticated Git and/or Helm repositories. An issue was discovered in `go-getter` library in versions prior to [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) that exposes SSH private keys in base64 format due to a failure in redacting such information from error messages. The vulnerable version of this library is used in Rancher through Fleet in versions of Fleet prior to [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9). This issue affects Rancher versions 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3. The issue was found and reported by Dagan Henderson from Raft Engineering. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | +| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | This vulnerability only affects customers using [Continuous Delivery with Fleet](../../integrations-in-rancher/fleet-gitops-at-scale/fleet-gitops-at-scale.md) for continuous delivery with authenticated Git and/or Helm repositories. An issue was discovered in `go-getter` library in versions prior to [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) that exposes SSH private keys in base64 format due to a failure in redacting such information from error messages. The vulnerable version of this library is used in Rancher through Fleet in versions of Fleet prior to [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9). This issue affects Rancher versions 2.5.0 up to and including 2.5.12 and from 2.6.0 up to and including 2.6.3. The issue was found and reported by Dagan Henderson from Raft Engineering. | 14 Apr 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) and [Rancher v2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | | [CVE-2021-36778](https://github.com/rancher/rancher/security/advisories/GHSA-4fc7-hc63-7fjg) | A vulnerability was discovered in Rancher versions from 2.5.0 up to and including 2.5.11 and from 2.6.0 up to and including 2.6.2, where an insufficient check of the same-origin policy when downloading Helm charts from a configured private repository can lead to exposure of the repository credentials to a third-party provider. This issue only happens when the user configures access credentials to a private repository in Rancher inside `Apps & Marketplace > Repositories`. The issue was found and reported by Martin Andreas Ullrich. | 14 Apr 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) and [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) | | [GHSA-hwm2-4ph6-w6m5](https://github.com/rancher/rancher/security/advisories/GHSA-hwm2-4ph6-w6m5) | A vulnerability was discovered in versions of Rancher starting 2.0 up to and including 2.6.3. The `restricted` pod security policy (PSP) provided in Rancher deviated from the upstream `restricted` policy provided in Kubernetes on account of which Rancher's PSP had `runAsUser` set to `runAsAny`, while upstream had `runAsUser` set to `MustRunAsNonRoot`. This allowed containers to run as any user, including a privileged user (`root`), even when Rancher's `restricted` policy was enforced on a project or at the cluster level. | 31 Mar 2022 | [Rancher v2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) | | [CVE-2021-36775](https://github.com/rancher/rancher/security/advisories/GHSA-28g7-896h-695v) | A vulnerability was discovered in Rancher versions up to and including 2.4.17, 2.5.11 and 2.6.2. After removing a `Project Role` associated with a group from the project, the bindings that granted access to cluster-scoped resources for those subjects were not deleted. This was due to an incomplete authorization logic check. A user who was a member of the affected group with authenticated access to Rancher could exploit this vulnerability to access resources they shouldn't have had access to. The exposure level would depend on the original permission level granted to the affected project role. This vulnerability only affected customers using group based authentication in Rancher. | 31 Mar 2022 | [Rancher v2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3), [Rancher v2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) and [Rancher v2.4.18](https://github.com/rancher/rancher/releases/tag/v2.4.18) | From 47b6db591854be5c2320391f921aa1b44fd7a388 Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Mon, 8 Apr 2024 19:06:44 -0700 Subject: [PATCH 32/67] Existing redirect target removed in this PR --- docusaurus.config.js | 18 +++--------------- 1 file changed, 3 insertions(+), 15 deletions(-) diff --git a/docusaurus.config.js b/docusaurus.config.js index e39fa77d9fe..938b3a79b3a 100644 --- a/docusaurus.config.js +++ b/docusaurus.config.js @@ -247,15 +247,15 @@ module.exports = { redirects: [ { // Redirects for multi-cluster apps removal (rancher-docs/issues/734) to: '/integrations-in-rancher/fleet', - from: ['/how-to-guides/new-user-guides/deploy-apps-across-clusters','/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet','/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps'] + from: ['/pages-for-subheaders/deploy-apps-across-clusters', '/how-to-guides/new-user-guides/deploy-apps-across-clusters', '/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet', '/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps'] }, { to: '/v2.8/integrations-in-rancher/fleet', - from: ['/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters','/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet','/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps'] + from: ['/v2.8/pages-for-subheaders/deploy-apps-across-clusters', '/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters', '/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet', '/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps'] }, { to: '/v2.7/integrations-in-rancher/fleet-gitops-at-scale', - from: ['/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters','/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet','/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps'] + from: ['/v2.7/pages-for-subheaders/deploy-apps-across-clusters', '/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters', '/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet', '/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/multi-cluster-apps'] },// Redirects for multi-cluster apps removal (rancher-docs/issues/734) (end) { to: '/faq/deprecated-features', @@ -1461,10 +1461,6 @@ module.exports = { to: '/v2.7/integrations-in-rancher/logging/custom-resource-configuration', from: '/v2.7/pages-for-subheaders/custom-resource-configuration' }, - { - to: '/v2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters', - from: '/v2.7/pages-for-subheaders/deploy-apps-across-clusters' - }, { to: '/v2.7/getting-started/quick-start-guides/deploy-rancher-manager', from: '/v2.7/pages-for-subheaders/deploy-rancher-manager' @@ -1813,10 +1809,6 @@ module.exports = { to: '/v2.8/integrations-in-rancher/logging/custom-resource-configuration', from: '/v2.8/pages-for-subheaders/custom-resource-configuration' }, - { - to: '/v2.8/how-to-guides/new-user-guides/deploy-apps-across-clusters', - from: '/v2.8/pages-for-subheaders/deploy-apps-across-clusters' - }, { to: '/v2.8/getting-started/quick-start-guides/deploy-rancher-manager', from: '/v2.8/pages-for-subheaders/deploy-rancher-manager' @@ -2161,10 +2153,6 @@ module.exports = { to: '/integrations-in-rancher/logging/custom-resource-configuration', from: '/pages-for-subheaders/custom-resource-configuration' }, - { - to: '/how-to-guides/new-user-guides/deploy-apps-across-clusters', - from: '/pages-for-subheaders/deploy-apps-across-clusters' - }, { to: '/getting-started/quick-start-guides/deploy-rancher-manager', from: '/pages-for-subheaders/deploy-rancher-manager' From 2105fc4c236622f900ece08ddadeb2f54b0eab30 Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Tue, 9 Apr 2024 09:11:04 -0700 Subject: [PATCH 33/67] Apply suggestions from code review Co-authored-by: Marty Hernandez Avedon --- .../enable-experimental-features/continuous-delivery.md | 2 +- .../enable-experimental-features/continuous-delivery.md | 2 +- .../enable-experimental-features/continuous-delivery.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md b/docs/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md index 1ea0e2eb2ea..1f56c890f43 100644 --- a/docs/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md +++ b/docs/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md @@ -6,7 +6,7 @@ title: Continuous Delivery -[Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet/fleet.md) comes preinstalled in Rancher can't be fully disabled. comes preinstalled in Rancher can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. +[Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet/fleet.md) comes preinstalled in Rancher and can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. To enable or disable this feature, refer to the instructions on [the main page about enabling experimental features.](enable-experimental-features.md) diff --git a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md index 8446287f45f..02930f889a6 100644 --- a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md +++ b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md @@ -6,7 +6,7 @@ title: Continuous Delivery -[Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet-gitops-at-scale/fleet-gitops-at-scale.md) comes preinstalled in Rancher can't be fully disabled. comes preinstalled in Rancher can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. +[Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet-gitops-at-scale/fleet-gitops-at-scale.md) comes preinstalled in Rancher and can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. To enable or disable this feature, refer to the instructions on [the main page about enabling experimental features.](enable-experimental-features.md) diff --git a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md index 1ea0e2eb2ea..1f56c890f43 100644 --- a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md +++ b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md @@ -6,7 +6,7 @@ title: Continuous Delivery -[Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet/fleet.md) comes preinstalled in Rancher can't be fully disabled. comes preinstalled in Rancher can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. +[Continuous Delivery with Fleet](../../../integrations-in-rancher/fleet/fleet.md) comes preinstalled in Rancher and can't be fully disabled. However, the Fleet feature for GitOps continuous delivery may be disabled using the `continuous-delivery` feature flag. To enable or disable this feature, refer to the instructions on [the main page about enabling experimental features.](enable-experimental-features.md) From 6ccd0b7b6c90ff22cd4f5eb8174940e47699f64d Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Tue, 9 Apr 2024 17:10:29 -0400 Subject: [PATCH 34/67] Syncing sidebar with page titles and disambiguating brief titles (#1197) * Syncing sidebar labels with page titles Deploying Rancher Server: Update sidebar label to match title * Installing/Upgrading Rancher: Update title to match sidebar This is a reference/hub page for install guides with no step-by-step instructions, so we're breaking the -ing rule to match other reference pages as well as the current sidebar label * Cluster Access: Update title to match sidebar * Kubernetes Persistent Storage: Volumes and Storage Classes - Update title to match sidebar * Don't have a Kubernetes cluster? Try one of these tutorials: Update title to match sidebar and make old title intro to page * Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials: Update title to match sidebar and make old title intro to page * versioning Deploying Rancher Server update to other sidebars * Setting Up Kubernetes Clusters in Rancher: Update sidebars to match title and other sidebar labels * capitalization * Creating a vSphere Cluster: Update sidebar to match title and other labels * Creating a Nutanix AOS Cluster: Update sidebar to match title and other labels * Kubernetes Clusters in Rancher Setup across the board for title and sidebar, to match convention in sidebar * Kubernetes Resources: Updated title to match sidebar and distinguish from identically-titled page in troubleshooting section * The Horizontal Pod Autoscaler: Updated title to match sidebar * Backups and Disaster Recovery: Update title to match sidebar * typo fix * revert to Installation and Upgrade of Rancher fix typo in title: Create Kubernetes Persistent files * fix typo in Persistent Storage files * Configuration: Update title to match sidebar item Monitoring V2 Configuration Guides * Setup Guide: Make both sidebar + title Istio Setup guides to match other sidebar labels * Best Practices: Update both to Best Practice Guides * Architecture: Update to match sidebar Rancher Architecture. Note that there are multiple pages with identical titles, one is on Fleet and another on some other subject * Architecture: Retitle logging-architecture.md files Logging Architecture * Architecture: Retitle fleet/architecture.md files Fleet Architecture * GKE Cluster Configuration: Update sidebar to match title and other labels in same section * Security: Update both to Rancher Security Guides * RKE Hardening Guide: Update to match sidebar * typo * RKE2 Hardening Guide: Update to match sidebar * K3s Hardening Guide: Update to match sidebar * various FAQ pages: Add FAQ to title to disambiguate content * Cloud Native Storage with Longhorn: Versioning so older pages match current title * rm international pages for now * typo in metadata killed build * updated sidebar: plural Istio Setup Guides * updating Monitoring Config Guides title/label and distinguishing from similar section under References * monitoring V2 config examples: rm 'V2' * Kubernetes Cluster Setup > Setting up a Kubernetes Cluster for Rancher Server --- docs/faq/dockershim.md | 2 +- docs/faq/security.md | 2 +- docs/faq/technical-items.md | 2 +- docs/faq/telemetry.md | 2 +- .../istio-setup-guide/istio-setup-guide.md | 2 +- .../monitoring-v2-configuration-guides.md | 2 +- .../about-rke1-templates.md | 2 +- .../backup-restore-and-disaster-recovery.md | 2 +- .../infrastructure-setup.md | 5 +++-- .../kubernetes-cluster-setup.md | 4 +++- .../kubernetes-clusters-in-rancher-setup.md | 2 +- .../horizontal-pod-autoscaler.md | 2 +- .../kubernetes-resources-setup.md | 2 +- .../load-balancer-and-ingress-controller.md | 2 +- .../access-clusters/access-clusters.md | 2 +- .../create-kubernetes-persistent-storage.md | 2 +- .../fleet/architecture.md | 2 +- .../logging/logging-architecture.md | 2 +- .../best-practices/best-practices.md | 2 +- .../monitoring-v2-configuration/examples.md | 2 +- .../monitoring-v2-configuration.md | 2 +- .../rancher-manager-architecture.md | 2 +- .../k3s-hardening-guide.md | 2 +- .../rke1-hardening-guide.md | 2 +- .../rke2-hardening-guide.md | 2 +- .../rancher-security/rancher-security.md | 2 +- sidebars.js | 20 +++++++++---------- .../version-2.0-2.4/faq/security.md | 2 +- .../version-2.0-2.4/faq/technical-items.md | 2 +- .../version-2.0-2.4/faq/telemetry.md | 2 +- .../about-rke1-templates.md | 2 +- .../istio-setup-guide/istio-setup-guide.md | 2 +- .../access-clusters/access-clusters.md | 2 +- .../create-kubernetes-persistent-storage.md | 2 +- .../backup-restore-and-disaster-recovery.md | 2 +- .../infrastructure-setup.md | 6 ++++-- .../kubernetes-cluster-setup.md | 4 +++- .../kubernetes-clusters-in-rancher-setup.md | 2 +- .../horizontal-pod-autoscaler.md | 2 +- .../kubernetes-resources-setup.md | 2 +- .../load-balancer-and-ingress-controller.md | 2 +- .../best-practices/best-practices.md | 2 +- .../rancher-manager-architecture.md | 2 +- .../rancher-security/rancher-security.md | 2 +- .../fleet-gitops-at-scale/architecture.md | 2 +- .../logging/logging-architecture.md | 2 +- .../integrations-in-rancher/longhorn.md | 2 +- versioned_docs/version-2.5/faq/security.md | 2 +- .../version-2.5/faq/technical-items.md | 2 +- versioned_docs/version-2.5/faq/telemetry.md | 2 +- .../about-rke1-templates.md | 2 +- .../istio-setup-guide/istio-setup-guide.md | 2 +- .../access-clusters/access-clusters.md | 2 +- .../create-kubernetes-persistent-storage.md | 2 +- .../monitoring-v2-configuration-guides.md | 2 +- .../backup-restore-and-disaster-recovery.md | 2 +- .../infrastructure-setup.md | 6 ++++-- .../kubernetes-cluster-setup.md | 4 +++- .../kubernetes-clusters-in-rancher-setup.md | 2 +- .../horizontal-pod-autoscaler.md | 2 +- .../kubernetes-resources-setup.md | 2 +- .../load-balancer-and-ingress-controller.md | 2 +- .../best-practices/best-practices.md | 2 +- .../rancher-manager-architecture.md | 2 +- .../rancher-security/rancher-security.md | 2 +- versioned_docs/version-2.6/faq/dockershim.md | 2 +- versioned_docs/version-2.6/faq/security.md | 2 +- .../version-2.6/faq/technical-items.md | 2 +- versioned_docs/version-2.6/faq/telemetry.md | 2 +- .../istio-setup-guide/istio-setup-guide.md | 2 +- .../monitoring-v2-configuration-guides.md | 2 +- .../about-rke1-templates.md | 2 +- .../backup-restore-and-disaster-recovery.md | 2 +- .../infrastructure-setup.md | 6 ++++-- .../kubernetes-cluster-setup.md | 4 +++- .../kubernetes-clusters-in-rancher-setup.md | 2 +- .../horizontal-pod-autoscaler.md | 2 +- .../kubernetes-resources-setup.md | 2 +- .../load-balancer-and-ingress-controller.md | 2 +- .../access-clusters/access-clusters.md | 2 +- .../create-kubernetes-persistent-storage.md | 2 +- .../fleet-gitops-at-scale/architecture.md | 2 +- .../logging/logging-architecture.md | 2 +- .../integrations-in-rancher/longhorn.md | 2 +- .../best-practices/best-practices.md | 2 +- .../monitoring-v2-configuration/examples.md | 2 +- .../monitoring-v2-configuration.md | 2 +- .../rancher-manager-architecture.md | 2 +- .../rancher-security/rancher-security.md | 2 +- versioned_docs/version-2.7/faq/dockershim.md | 2 +- versioned_docs/version-2.7/faq/security.md | 2 +- .../version-2.7/faq/technical-items.md | 2 +- versioned_docs/version-2.7/faq/telemetry.md | 2 +- .../istio-setup-guide/istio-setup-guide.md | 2 +- .../monitoring-v2-configuration-guides.md | 2 +- .../about-rke1-templates.md | 2 +- .../backup-restore-and-disaster-recovery.md | 2 +- .../infrastructure-setup.md | 6 ++++-- .../kubernetes-cluster-setup.md | 4 +++- .../kubernetes-clusters-in-rancher-setup.md | 2 +- .../horizontal-pod-autoscaler.md | 2 +- .../kubernetes-resources-setup.md | 2 +- .../load-balancer-and-ingress-controller.md | 2 +- .../access-clusters/access-clusters.md | 2 +- .../create-kubernetes-persistent-storage.md | 2 +- .../fleet-gitops-at-scale/architecture.md | 2 +- .../logging/logging-architecture.md | 2 +- .../integrations-in-rancher/longhorn.md | 2 +- .../best-practices/best-practices.md | 2 +- .../monitoring-v2-configuration/examples.md | 2 +- .../monitoring-v2-configuration.md | 2 +- .../rancher-manager-architecture.md | 2 +- .../k3s-hardening-guide.md | 2 +- .../rke1-hardening-guide.md | 2 +- .../rke2-hardening-guide.md | 2 +- .../rancher-security/rancher-security.md | 3 ++- versioned_docs/version-2.8/faq/dockershim.md | 2 +- versioned_docs/version-2.8/faq/security.md | 2 +- .../version-2.8/faq/technical-items.md | 2 +- versioned_docs/version-2.8/faq/telemetry.md | 2 +- .../istio-setup-guide/istio-setup-guide.md | 2 +- .../monitoring-v2-configuration-guides.md | 2 +- .../about-rke1-templates.md | 2 +- .../backup-restore-and-disaster-recovery.md | 2 +- .../infrastructure-setup.md | 6 ++++-- .../kubernetes-cluster-setup.md | 4 +++- .../kubernetes-clusters-in-rancher-setup.md | 2 +- .../horizontal-pod-autoscaler.md | 2 +- .../kubernetes-resources-setup.md | 2 +- .../load-balancer-and-ingress-controller.md | 2 +- .../access-clusters/access-clusters.md | 2 +- .../create-kubernetes-persistent-storage.md | 2 +- .../fleet/architecture.md | 2 +- .../logging/logging-architecture.md | 2 +- .../best-practices/best-practices.md | 2 +- .../monitoring-v2-configuration/examples.md | 2 +- .../monitoring-v2-configuration.md | 2 +- .../rancher-manager-architecture.md | 2 +- .../k3s-hardening-guide.md | 2 +- .../rke1-hardening-guide.md | 2 +- .../rke2-hardening-guide.md | 2 +- .../rancher-security/rancher-security.md | 2 +- .../version-2.0-2.4-sidebars.json | 12 +++++------ versioned_sidebars/version-2.5-sidebars.json | 16 +++++++-------- versioned_sidebars/version-2.6-sidebars.json | 20 +++++++++---------- versioned_sidebars/version-2.7-sidebars.json | 20 +++++++++---------- versioned_sidebars/version-2.8-sidebars.json | 20 +++++++++---------- 147 files changed, 225 insertions(+), 201 deletions(-) diff --git a/docs/faq/dockershim.md b/docs/faq/dockershim.md index 23b982f7b30..4e710c9746b 100644 --- a/docs/faq/dockershim.md +++ b/docs/faq/dockershim.md @@ -1,5 +1,5 @@ --- -title: Dockershim +title: Dockershim FAQ --- diff --git a/docs/faq/security.md b/docs/faq/security.md index 684444e7303..46f774bf722 100644 --- a/docs/faq/security.md +++ b/docs/faq/security.md @@ -1,5 +1,5 @@ --- -title: Security +title: Security FAQ --- diff --git a/docs/faq/technical-items.md b/docs/faq/technical-items.md index db2500b7fbf..42bfe966726 100644 --- a/docs/faq/technical-items.md +++ b/docs/faq/technical-items.md @@ -1,5 +1,5 @@ --- -title: Technical +title: Technical FAQ --- diff --git a/docs/faq/telemetry.md b/docs/faq/telemetry.md index a510347e556..edfcaebed4e 100644 --- a/docs/faq/telemetry.md +++ b/docs/faq/telemetry.md @@ -1,5 +1,5 @@ --- -title: Telemetry +title: Telemetry FAQ --- diff --git a/docs/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md b/docs/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md index cbdeee0602b..cb01f789246 100644 --- a/docs/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md +++ b/docs/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md @@ -1,5 +1,5 @@ --- -title: Setup Guide +title: Istio Setup Guides --- diff --git a/docs/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md b/docs/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md index 9fa587aef2b..e3de2cf69cf 100644 --- a/docs/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md +++ b/docs/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md @@ -1,5 +1,5 @@ --- -title: Configuration +title: Monitoring Configuration Guides --- diff --git a/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md b/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md index 228ec4c0992..e67a2da5d6d 100644 --- a/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md +++ b/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md @@ -1,5 +1,5 @@ --- -title: RKE Templates +title: About RKE1 Templates --- diff --git a/docs/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md b/docs/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md index c5afbae3842..71467ac7403 100644 --- a/docs/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md +++ b/docs/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md @@ -1,5 +1,5 @@ --- -title: Backups and Disaster Recovery +title: Backup, Restore, and Disaster Recovery keywords: [rancher backup restore, rancher backup and restore, backup restore rancher, rancher backup and restore rancher] --- diff --git a/docs/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md b/docs/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md index 7a79e5f5622..19ee80b9594 100644 --- a/docs/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md +++ b/docs/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md @@ -1,12 +1,13 @@ --- -title: Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials. +title: Infrastructure Setup --- -To set up infrastructure for a high-availability K3s Kubernetes cluster with an external DB, refer to [this page.](ha-k3s-kubernetes-cluster.md) +Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials. +To set up infrastructure for a high-availability K3s Kubernetes cluster with an external database, refer to [this page.](ha-k3s-kubernetes-cluster.md) To set up infrastructure for a high-availability RKE Kubernetes cluster, refer to [this page.](ha-rke1-kubernetes-cluster.md) diff --git a/docs/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md b/docs/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md index 00c22ead1c3..05267457f70 100644 --- a/docs/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md +++ b/docs/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md @@ -1,11 +1,13 @@ --- -title: "Don't have a Kubernetes cluster? Try one of these tutorials." +title: Setting up a Kubernetes Cluster for Rancher Server --- +Don't have a Kubernetes cluster? Try one of these tutorials. + This section contains information on how to install a Kubernetes cluster that the Rancher server can be installed on. Rancher can run on any Kubernetes cluster. diff --git a/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md b/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md index 230a3d6212b..fcacb555608 100644 --- a/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md +++ b/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md @@ -1,5 +1,5 @@ --- -title: Setting up Kubernetes Clusters in Rancher +title: Kubernetes Clusters in Rancher Setup description: Provisioning Kubernetes Clusters --- diff --git a/docs/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md b/docs/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md index 8d084cb00d7..bacdd796fce 100644 --- a/docs/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md +++ b/docs/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md @@ -1,5 +1,5 @@ --- -title: The Horizontal Pod Autoscaler +title: Horizontal Pod Autoscaler description: Learn about the horizontal pod autoscaler (HPA). How to manage HPAs and how to test them with a service deployment --- diff --git a/docs/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md b/docs/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md index 7879660225d..3e97882659c 100644 --- a/docs/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md +++ b/docs/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md @@ -1,5 +1,5 @@ --- -title: Kubernetes Resources +title: Kubernetes Resources Setup --- diff --git a/docs/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md b/docs/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md index 934a552f019..48d3f52e276 100644 --- a/docs/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md +++ b/docs/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md @@ -1,5 +1,5 @@ --- -title: Set Up Load Balancer and Ingress Controller within Rancher +title: Load Balancer and Ingress Controller Setup within Rancher description: Learn how you can set up load balancers and ingress controllers to redirect service requests within Rancher, and learn about the limitations of load balancers --- diff --git a/docs/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md b/docs/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md index ca181ffc07a..66a81355f0e 100644 --- a/docs/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md +++ b/docs/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md @@ -1,5 +1,5 @@ --- -title: Cluster Access +title: Access Clusters --- diff --git a/docs/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/docs/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md index 8b2d0db56c7..7ada9cd85af 100644 --- a/docs/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md +++ b/docs/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -1,5 +1,5 @@ --- -title: "Kubernetes Persistent Storage: Volumes and Storage Classes" +title: Create Kubernetes Persistent Volumes and Storage Classes description: "Learn about the two ways with which you can create persistent storage in Kubernetes: persistent volumes and storage classes" --- diff --git a/docs/integrations-in-rancher/fleet/architecture.md b/docs/integrations-in-rancher/fleet/architecture.md index dcdd07ce36f..556bd33f4e8 100644 --- a/docs/integrations-in-rancher/fleet/architecture.md +++ b/docs/integrations-in-rancher/fleet/architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Fleet Architecture --- diff --git a/docs/integrations-in-rancher/logging/logging-architecture.md b/docs/integrations-in-rancher/logging/logging-architecture.md index 958bc5d3069..f4b716a6c2e 100644 --- a/docs/integrations-in-rancher/logging/logging-architecture.md +++ b/docs/integrations-in-rancher/logging/logging-architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Logging Architecture --- diff --git a/docs/reference-guides/best-practices/best-practices.md b/docs/reference-guides/best-practices/best-practices.md index d2520a3e785..c0b4a9cc233 100644 --- a/docs/reference-guides/best-practices/best-practices.md +++ b/docs/reference-guides/best-practices/best-practices.md @@ -1,5 +1,5 @@ --- -title: Best Practices Guide +title: Best Practice Guides --- diff --git a/docs/reference-guides/monitoring-v2-configuration/examples.md b/docs/reference-guides/monitoring-v2-configuration/examples.md index d6ef1920256..1660d70a31f 100644 --- a/docs/reference-guides/monitoring-v2-configuration/examples.md +++ b/docs/reference-guides/monitoring-v2-configuration/examples.md @@ -1,5 +1,5 @@ --- -title: Monitoring V2 Configuration Examples +title: Monitoring Configuration Examples --- diff --git a/docs/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md b/docs/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md index 7fea6f9a7d9..27b0c7ff364 100644 --- a/docs/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md +++ b/docs/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md @@ -1,5 +1,5 @@ --- -title: Monitoring V2 Configuration +title: Monitoring Configuration Reference --- diff --git a/docs/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md b/docs/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md index 0e01bd3d293..1a0f0f0e5f8 100644 --- a/docs/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md +++ b/docs/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Rancher Architecture --- diff --git a/docs/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md b/docs/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md index 88f00624b20..90f9b494857 100644 --- a/docs/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md +++ b/docs/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md @@ -1,5 +1,5 @@ --- -title: K3s Hardening Guide +title: K3s Hardening Guides --- diff --git a/docs/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md b/docs/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md index c7992611f8a..22c465eb07d 100644 --- a/docs/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md +++ b/docs/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md @@ -1,5 +1,5 @@ --- -title: RKE Hardening Guide +title: RKE Hardening Guides --- diff --git a/docs/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md b/docs/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md index a69c97665bb..bf79c65bd58 100644 --- a/docs/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md +++ b/docs/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md @@ -1,5 +1,5 @@ --- -title: RKE2 Hardening Guide +title: RKE2 Hardening Guides --- diff --git a/docs/reference-guides/rancher-security/rancher-security.md b/docs/reference-guides/rancher-security/rancher-security.md index 2a6fed33f3d..f6d56c11654 100644 --- a/docs/reference-guides/rancher-security/rancher-security.md +++ b/docs/reference-guides/rancher-security/rancher-security.md @@ -1,5 +1,5 @@ --- -title: Security +title: Rancher Security Guides --- diff --git a/sidebars.js b/sidebars.js index a08a5054420..b432826abd6 100644 --- a/sidebars.js +++ b/sidebars.js @@ -35,7 +35,7 @@ const sidebars = { items: [ { type: 'category', - label: 'Deploying Rancher', + label: 'Deploying Rancher Server', link: { type: 'doc', id: "getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager", @@ -405,7 +405,7 @@ const sidebars = { }, { type: 'category', - label: 'Kubernetes Cluster Setup', + label: 'Setting up a Kubernetes Cluster for Rancher Server', link: { type: 'doc', id: "how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup", @@ -534,7 +534,7 @@ const sidebars = { "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster", { type: 'category', - label: 'vSphere', + label: 'Creating a vSphere Cluster', link: { type: 'doc', id: "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere", @@ -547,7 +547,7 @@ const sidebars = { }, { type: 'category', - label: 'Nutanix', + label: 'Creating a Nutanix AOS Cluster', link: { type: 'doc', id: "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix", @@ -723,7 +723,7 @@ const sidebars = { }, { type: 'category', - label: 'Monitoring V2 Configuration Guides', + label: 'Monitoring Configuration Guides', link: { type: 'doc', id: "how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides", @@ -746,7 +746,7 @@ const sidebars = { }, { type: 'category', - label: 'Istio Setup Guide', + label: 'Istio Setup Guides', link: { type: 'doc', id: "how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide", @@ -808,7 +808,7 @@ const sidebars = { items: [ { type: 'category', - label: 'Best Practices', + label: 'Best Practice Guides', link: { type: 'doc', id: "reference-guides/best-practices/best-practices", @@ -880,7 +880,7 @@ const sidebars = { "reference-guides/cluster-configuration/rancher-server-configuration/aks-cluster-configuration", { type: 'category', - label: 'GKE Cluster Configuration', + label: 'GKE Cluster Configuration Reference', link: { type: 'doc', id: "reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration", @@ -972,7 +972,7 @@ const sidebars = { "reference-guides/kubernetes-concepts", { type: 'category', - label: 'Monitoring V2 Configuration', + label: 'Monitoring Configuration Reference', link: { type: 'doc', id: "reference-guides/monitoring-v2-configuration/monitoring-v2-configuration", @@ -1043,7 +1043,7 @@ const sidebars = { "reference-guides/rancher-webhook", { type: 'category', - label: 'Rancher Security', + label: 'Rancher Security Guides', link: { type: 'doc', id: "reference-guides/rancher-security/rancher-security", diff --git a/versioned_docs/version-2.0-2.4/faq/security.md b/versioned_docs/version-2.0-2.4/faq/security.md index 2ce3873f7ee..9d23ec6455e 100644 --- a/versioned_docs/version-2.0-2.4/faq/security.md +++ b/versioned_docs/version-2.0-2.4/faq/security.md @@ -1,5 +1,5 @@ --- -title: Security +title: Security FAQ --- diff --git a/versioned_docs/version-2.0-2.4/faq/technical-items.md b/versioned_docs/version-2.0-2.4/faq/technical-items.md index fbbbce3bfdd..3cc8542d4a0 100644 --- a/versioned_docs/version-2.0-2.4/faq/technical-items.md +++ b/versioned_docs/version-2.0-2.4/faq/technical-items.md @@ -1,5 +1,5 @@ --- -title: Technical +title: Technical FAQ --- diff --git a/versioned_docs/version-2.0-2.4/faq/telemetry.md b/versioned_docs/version-2.0-2.4/faq/telemetry.md index a510347e556..edfcaebed4e 100644 --- a/versioned_docs/version-2.0-2.4/faq/telemetry.md +++ b/versioned_docs/version-2.0-2.4/faq/telemetry.md @@ -1,5 +1,5 @@ --- -title: Telemetry +title: Telemetry FAQ --- diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md b/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md index a0747eeb94a..d2402a130ab 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md @@ -1,5 +1,5 @@ --- -title: RKE Templates +title: About RKE1 Templates --- diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md b/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md index 59f6f649581..81a520e0d7e 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md @@ -1,5 +1,5 @@ --- -title: Setup Guide +title: Istio Setup Guides --- diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/manage-clusters/access-clusters/access-clusters.md b/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/manage-clusters/access-clusters/access-clusters.md index 0dc414e4b4f..0879cf322be 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/manage-clusters/access-clusters/access-clusters.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/manage-clusters/access-clusters/access-clusters.md @@ -1,5 +1,5 @@ --- -title: Cluster Access +title: Access Clusters --- diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md index 2d12df3b6f4..4707706acf4 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -1,5 +1,5 @@ --- -title: "Kubernetes Persistent Storage: Volumes and Storage Classes" +title: Create Kubernetes Persistent Volumes and Storage Classes description: "Learn about the two ways with which you can create persistent storage in Kubernetes: persistent volumes and storage classes" --- diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md index 44d6ac9a85c..f5e84c8b6d7 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md @@ -1,5 +1,5 @@ --- -title: Backups and Disaster Recovery +title: Backup, Restore, and Disaster Recovery keywords: [rancher v2.0-v2.4 backup restore, rancher v2.0-v2.4 backup and restore, backup restore rancher v2.0-v2.4, backup and restore rancher v2.0-v2.4] --- diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md index 7a79e5f5622..e361e6c2f02 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md @@ -1,12 +1,14 @@ --- -title: Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials. +title: Infrastructure Setup --- -To set up infrastructure for a high-availability K3s Kubernetes cluster with an external DB, refer to [this page.](ha-k3s-kubernetes-cluster.md) +Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials. + +To set up infrastructure for a high-availability K3s Kubernetes cluster with an external database, refer to [this page.](ha-k3s-kubernetes-cluster.md) To set up infrastructure for a high-availability RKE Kubernetes cluster, refer to [this page.](ha-rke1-kubernetes-cluster.md) diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md index 714755763ad..2687702d304 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md @@ -1,11 +1,13 @@ --- -title: "Don't have a Kubernetes cluster? Try one of these tutorials." +title: Setting up a Kubernetes Cluster for Rancher Server --- +Don't have a Kubernetes cluster? Try one of these tutorials. + This section contains information on how to install a Kubernetes cluster that the Rancher server can be installed on. In Rancher before v2.4, the Rancher server needed to run on an RKE Kubernetes cluster. diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md index 9778c4b0db7..d2ffb8e260d 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md @@ -1,5 +1,5 @@ --- -title: Setting up Kubernetes Clusters in Rancher +title: Kubernetes Clusters in Rancher Setup description: Provisioning Kubernetes Clusters --- diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md index d0d1b233373..2b9a134cbe2 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md @@ -1,5 +1,5 @@ --- -title: The Horizontal Pod Autoscaler +title: Horizontal Pod Autoscaler description: Learn about the horizontal pod autoscaler (HPA). How to manage HPAs and how to test them with a service deployment --- diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md index 1df605165d2..0fb58b07635 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md @@ -1,5 +1,5 @@ --- -title: Kubernetes Resources +title: Kubernetes Resources Setup --- diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md index 3e166096ded..2644cb6d942 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md @@ -1,5 +1,5 @@ --- -title: Set Up Load Balancer and Ingress Controller within Rancher +title: Load Balancer and Ingress Controller Setup within Rancher description: Learn how you can set up load balancers and ingress controllers to redirect service requests within Rancher, and learn about the limitations of load balancers --- diff --git a/versioned_docs/version-2.0-2.4/reference-guides/best-practices/best-practices.md b/versioned_docs/version-2.0-2.4/reference-guides/best-practices/best-practices.md index d2520a3e785..c0b4a9cc233 100644 --- a/versioned_docs/version-2.0-2.4/reference-guides/best-practices/best-practices.md +++ b/versioned_docs/version-2.0-2.4/reference-guides/best-practices/best-practices.md @@ -1,5 +1,5 @@ --- -title: Best Practices Guide +title: Best Practice Guides --- diff --git a/versioned_docs/version-2.0-2.4/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md b/versioned_docs/version-2.0-2.4/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md index a3df555f326..139bb93580c 100644 --- a/versioned_docs/version-2.0-2.4/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md +++ b/versioned_docs/version-2.0-2.4/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Rancher Architecture --- diff --git a/versioned_docs/version-2.0-2.4/reference-guides/rancher-security/rancher-security.md b/versioned_docs/version-2.0-2.4/reference-guides/rancher-security/rancher-security.md index 14bcde93599..c4dedb25e8d 100644 --- a/versioned_docs/version-2.0-2.4/reference-guides/rancher-security/rancher-security.md +++ b/versioned_docs/version-2.0-2.4/reference-guides/rancher-security/rancher-security.md @@ -1,5 +1,5 @@ --- -title: Security +title: Rancher Security Guides --- diff --git a/versioned_docs/version-2.5/explanations/integrations-in-rancher/fleet-gitops-at-scale/architecture.md b/versioned_docs/version-2.5/explanations/integrations-in-rancher/fleet-gitops-at-scale/architecture.md index 9d64e38de41..169b667a27e 100644 --- a/versioned_docs/version-2.5/explanations/integrations-in-rancher/fleet-gitops-at-scale/architecture.md +++ b/versioned_docs/version-2.5/explanations/integrations-in-rancher/fleet-gitops-at-scale/architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Fleet Architecture --- diff --git a/versioned_docs/version-2.5/explanations/integrations-in-rancher/logging/logging-architecture.md b/versioned_docs/version-2.5/explanations/integrations-in-rancher/logging/logging-architecture.md index 5c333557df5..2cb176749d4 100644 --- a/versioned_docs/version-2.5/explanations/integrations-in-rancher/logging/logging-architecture.md +++ b/versioned_docs/version-2.5/explanations/integrations-in-rancher/logging/logging-architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Logging Architecture --- diff --git a/versioned_docs/version-2.5/explanations/integrations-in-rancher/longhorn.md b/versioned_docs/version-2.5/explanations/integrations-in-rancher/longhorn.md index c80ffcb2fd6..31aa0179fc1 100644 --- a/versioned_docs/version-2.5/explanations/integrations-in-rancher/longhorn.md +++ b/versioned_docs/version-2.5/explanations/integrations-in-rancher/longhorn.md @@ -1,5 +1,5 @@ --- -title: Longhorn - Cloud native distributed block storage for Kubernetes +title: Cloud Native Storage with Longhorn --- diff --git a/versioned_docs/version-2.5/faq/security.md b/versioned_docs/version-2.5/faq/security.md index 2ce3873f7ee..9d23ec6455e 100644 --- a/versioned_docs/version-2.5/faq/security.md +++ b/versioned_docs/version-2.5/faq/security.md @@ -1,5 +1,5 @@ --- -title: Security +title: Security FAQ --- diff --git a/versioned_docs/version-2.5/faq/technical-items.md b/versioned_docs/version-2.5/faq/technical-items.md index b3bd375d0d3..7328c27c200 100644 --- a/versioned_docs/version-2.5/faq/technical-items.md +++ b/versioned_docs/version-2.5/faq/technical-items.md @@ -1,5 +1,5 @@ --- -title: Technical +title: Technical FAQ --- diff --git a/versioned_docs/version-2.5/faq/telemetry.md b/versioned_docs/version-2.5/faq/telemetry.md index a510347e556..edfcaebed4e 100644 --- a/versioned_docs/version-2.5/faq/telemetry.md +++ b/versioned_docs/version-2.5/faq/telemetry.md @@ -1,5 +1,5 @@ --- -title: Telemetry +title: Telemetry FAQ --- diff --git a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md index fa2826e95f5..39235a93292 100644 --- a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md +++ b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md @@ -1,5 +1,5 @@ --- -title: RKE Templates +title: About RKE1 Templates --- diff --git a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md index e8d1210fecc..cfee775160e 100644 --- a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md +++ b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md @@ -1,5 +1,5 @@ --- -title: Setup Guide +title: Istio Setup Guides --- diff --git a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/access-clusters/access-clusters.md b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/access-clusters/access-clusters.md index f44ee167ab4..9ad4ecbe30f 100644 --- a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/access-clusters/access-clusters.md +++ b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/access-clusters/access-clusters.md @@ -1,5 +1,5 @@ --- -title: Cluster Access +title: Access Clusters --- diff --git a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md index 96509884700..847e2eed848 100644 --- a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md +++ b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -1,5 +1,5 @@ --- -title: "Kubernetes Persistent Storage: Volumes and Storage Classes" +title: Create Kubernetes Persistent Volumes and Storage Classes description: "Learn about the two ways with which you can create persistent storage in Kubernetes: persistent volumes and storage classes" --- diff --git a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md index 91690613723..abcfdd14fa1 100644 --- a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md +++ b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md @@ -1,5 +1,5 @@ --- -title: Configuration +title: Monitoring V2 Configuration Guides --- diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md index eb07c014ac1..cc82cec195b 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md @@ -1,5 +1,5 @@ --- -title: Backups and Disaster Recovery +title: Backup, Restore, and Disaster Recovery keywords: [rancher v2.5 backup restore, rancher v2.5 backup and restore, backup restore rancher v2.5, backup and restore rancher v2.5] --- diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md index 7a79e5f5622..e361e6c2f02 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md @@ -1,12 +1,14 @@ --- -title: Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials. +title: Infrastructure Setup --- -To set up infrastructure for a high-availability K3s Kubernetes cluster with an external DB, refer to [this page.](ha-k3s-kubernetes-cluster.md) +Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials. + +To set up infrastructure for a high-availability K3s Kubernetes cluster with an external database, refer to [this page.](ha-k3s-kubernetes-cluster.md) To set up infrastructure for a high-availability RKE Kubernetes cluster, refer to [this page.](ha-rke1-kubernetes-cluster.md) diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md index 9a16092ead2..56783110778 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md @@ -1,11 +1,13 @@ --- -title: "Don't have a Kubernetes cluster? Try one of these tutorials." +title: Setting up a Kubernetes Cluster for Rancher Server --- +Don't have a Kubernetes cluster? Try one of these tutorials. + This section contains information on how to install a Kubernetes cluster that the Rancher server can be installed on. In Rancher v2.5, Rancher can run on any Kubernetes cluster. diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md index 76f7dbf8ea1..546ae7bdaf0 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md @@ -1,5 +1,5 @@ --- -title: Setting up Kubernetes Clusters in Rancher +title: Kubernetes Clusters in Rancher Setup description: Provisioning Kubernetes Clusters --- diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md index 9e1073555d1..415a7ab68a8 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md @@ -1,5 +1,5 @@ --- -title: The Horizontal Pod Autoscaler +title: Horizontal Pod Autoscaler description: Learn about the horizontal pod autoscaler (HPA). How to manage HPAs and how to test them with a service deployment --- diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md index 990a8efa844..1a3d0f24fd4 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md @@ -1,5 +1,5 @@ --- -title: Kubernetes Resources +title: Kubernetes Resources Setup --- diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md index 6163d2d99b5..0f38e9ad7e0 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md @@ -1,5 +1,5 @@ --- -title: Set Up Load Balancer and Ingress Controller within Rancher +title: Load Balancer and Ingress Controller Setup within Rancher description: Learn how you can set up load balancers and ingress controllers to redirect service requests within Rancher, and learn about the limitations of load balancers --- diff --git a/versioned_docs/version-2.5/reference-guides/best-practices/best-practices.md b/versioned_docs/version-2.5/reference-guides/best-practices/best-practices.md index d2520a3e785..c0b4a9cc233 100644 --- a/versioned_docs/version-2.5/reference-guides/best-practices/best-practices.md +++ b/versioned_docs/version-2.5/reference-guides/best-practices/best-practices.md @@ -1,5 +1,5 @@ --- -title: Best Practices Guide +title: Best Practice Guides --- diff --git a/versioned_docs/version-2.5/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md b/versioned_docs/version-2.5/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md index 4f62034f2b8..a6781ca5a10 100644 --- a/versioned_docs/version-2.5/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md +++ b/versioned_docs/version-2.5/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Rancher Architecture --- diff --git a/versioned_docs/version-2.5/reference-guides/rancher-security/rancher-security.md b/versioned_docs/version-2.5/reference-guides/rancher-security/rancher-security.md index 2027340c382..5ca30e89ffc 100644 --- a/versioned_docs/version-2.5/reference-guides/rancher-security/rancher-security.md +++ b/versioned_docs/version-2.5/reference-guides/rancher-security/rancher-security.md @@ -1,5 +1,5 @@ --- -title: Security +title: Rancher Security Guides --- diff --git a/versioned_docs/version-2.6/faq/dockershim.md b/versioned_docs/version-2.6/faq/dockershim.md index 23b982f7b30..4e710c9746b 100644 --- a/versioned_docs/version-2.6/faq/dockershim.md +++ b/versioned_docs/version-2.6/faq/dockershim.md @@ -1,5 +1,5 @@ --- -title: Dockershim +title: Dockershim FAQ --- diff --git a/versioned_docs/version-2.6/faq/security.md b/versioned_docs/version-2.6/faq/security.md index 684444e7303..46f774bf722 100644 --- a/versioned_docs/version-2.6/faq/security.md +++ b/versioned_docs/version-2.6/faq/security.md @@ -1,5 +1,5 @@ --- -title: Security +title: Security FAQ --- diff --git a/versioned_docs/version-2.6/faq/technical-items.md b/versioned_docs/version-2.6/faq/technical-items.md index db2500b7fbf..42bfe966726 100644 --- a/versioned_docs/version-2.6/faq/technical-items.md +++ b/versioned_docs/version-2.6/faq/technical-items.md @@ -1,5 +1,5 @@ --- -title: Technical +title: Technical FAQ --- diff --git a/versioned_docs/version-2.6/faq/telemetry.md b/versioned_docs/version-2.6/faq/telemetry.md index a510347e556..edfcaebed4e 100644 --- a/versioned_docs/version-2.6/faq/telemetry.md +++ b/versioned_docs/version-2.6/faq/telemetry.md @@ -1,5 +1,5 @@ --- -title: Telemetry +title: Telemetry FAQ --- diff --git a/versioned_docs/version-2.6/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md b/versioned_docs/version-2.6/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md index cbdeee0602b..cb01f789246 100644 --- a/versioned_docs/version-2.6/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md +++ b/versioned_docs/version-2.6/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md @@ -1,5 +1,5 @@ --- -title: Setup Guide +title: Istio Setup Guides --- diff --git a/versioned_docs/version-2.6/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md b/versioned_docs/version-2.6/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md index b4ae1e536e5..aa81f4f1e74 100644 --- a/versioned_docs/version-2.6/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md +++ b/versioned_docs/version-2.6/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md @@ -1,5 +1,5 @@ --- -title: Configuration +title: Monitoring Configuration Guides --- diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md index 228ec4c0992..e67a2da5d6d 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md @@ -1,5 +1,5 @@ --- -title: RKE Templates +title: About RKE1 Templates --- diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md index 3fb275389dc..4b496b11c35 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md @@ -1,5 +1,5 @@ --- -title: Backups and Disaster Recovery +title: Backup, Restore, and Disaster Recovery keywords: [rancher v2.6 backup restore, rancher v2.6 backup and restore, backup restore rancher v2.6, backup and restore rancher v2.6] --- diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md index 7a79e5f5622..e361e6c2f02 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md @@ -1,12 +1,14 @@ --- -title: Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials. +title: Infrastructure Setup --- -To set up infrastructure for a high-availability K3s Kubernetes cluster with an external DB, refer to [this page.](ha-k3s-kubernetes-cluster.md) +Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials. + +To set up infrastructure for a high-availability K3s Kubernetes cluster with an external database, refer to [this page.](ha-k3s-kubernetes-cluster.md) To set up infrastructure for a high-availability RKE Kubernetes cluster, refer to [this page.](ha-rke1-kubernetes-cluster.md) diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md index 00c22ead1c3..05267457f70 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md @@ -1,11 +1,13 @@ --- -title: "Don't have a Kubernetes cluster? Try one of these tutorials." +title: Setting up a Kubernetes Cluster for Rancher Server --- +Don't have a Kubernetes cluster? Try one of these tutorials. + This section contains information on how to install a Kubernetes cluster that the Rancher server can be installed on. Rancher can run on any Kubernetes cluster. diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md index 230a3d6212b..fcacb555608 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md @@ -1,5 +1,5 @@ --- -title: Setting up Kubernetes Clusters in Rancher +title: Kubernetes Clusters in Rancher Setup description: Provisioning Kubernetes Clusters --- diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md index 8d084cb00d7..bacdd796fce 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md @@ -1,5 +1,5 @@ --- -title: The Horizontal Pod Autoscaler +title: Horizontal Pod Autoscaler description: Learn about the horizontal pod autoscaler (HPA). How to manage HPAs and how to test them with a service deployment --- diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md index 74ece7fc40c..53c50d05855 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md @@ -1,5 +1,5 @@ --- -title: Kubernetes Resources +title: Kubernetes Resources Setup --- diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md index 934a552f019..48d3f52e276 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md @@ -1,5 +1,5 @@ --- -title: Set Up Load Balancer and Ingress Controller within Rancher +title: Load Balancer and Ingress Controller Setup within Rancher description: Learn how you can set up load balancers and ingress controllers to redirect service requests within Rancher, and learn about the limitations of load balancers --- diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md index a1f9647729f..57acff99a8f 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md @@ -1,5 +1,5 @@ --- -title: Cluster Access +title: Access Clusters --- diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md index b7f9af5c766..a3d842f5265 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -1,5 +1,5 @@ --- -title: "Kubernetes Persistent Storage: Volumes and Storage Classes" +title: Create Kubernetes Persistent Volumes and Storage Classes description: "Learn about the two ways with which you can create persistent storage in Kubernetes: persistent volumes and storage classes" --- diff --git a/versioned_docs/version-2.6/integrations-in-rancher/fleet-gitops-at-scale/architecture.md b/versioned_docs/version-2.6/integrations-in-rancher/fleet-gitops-at-scale/architecture.md index 9d64e38de41..169b667a27e 100644 --- a/versioned_docs/version-2.6/integrations-in-rancher/fleet-gitops-at-scale/architecture.md +++ b/versioned_docs/version-2.6/integrations-in-rancher/fleet-gitops-at-scale/architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Fleet Architecture --- diff --git a/versioned_docs/version-2.6/integrations-in-rancher/logging/logging-architecture.md b/versioned_docs/version-2.6/integrations-in-rancher/logging/logging-architecture.md index f6c71cdc27a..328418b4798 100644 --- a/versioned_docs/version-2.6/integrations-in-rancher/logging/logging-architecture.md +++ b/versioned_docs/version-2.6/integrations-in-rancher/logging/logging-architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Logging Architecture --- diff --git a/versioned_docs/version-2.6/integrations-in-rancher/longhorn.md b/versioned_docs/version-2.6/integrations-in-rancher/longhorn.md index b41e950efb9..1a2806ecc09 100644 --- a/versioned_docs/version-2.6/integrations-in-rancher/longhorn.md +++ b/versioned_docs/version-2.6/integrations-in-rancher/longhorn.md @@ -1,5 +1,5 @@ --- -title: Longhorn - Cloud native distributed block storage for Kubernetes +title: Cloud Native Storage with Longhorn --- diff --git a/versioned_docs/version-2.6/reference-guides/best-practices/best-practices.md b/versioned_docs/version-2.6/reference-guides/best-practices/best-practices.md index d2520a3e785..c0b4a9cc233 100644 --- a/versioned_docs/version-2.6/reference-guides/best-practices/best-practices.md +++ b/versioned_docs/version-2.6/reference-guides/best-practices/best-practices.md @@ -1,5 +1,5 @@ --- -title: Best Practices Guide +title: Best Practice Guides --- diff --git a/versioned_docs/version-2.6/reference-guides/monitoring-v2-configuration/examples.md b/versioned_docs/version-2.6/reference-guides/monitoring-v2-configuration/examples.md index d6ef1920256..1660d70a31f 100644 --- a/versioned_docs/version-2.6/reference-guides/monitoring-v2-configuration/examples.md +++ b/versioned_docs/version-2.6/reference-guides/monitoring-v2-configuration/examples.md @@ -1,5 +1,5 @@ --- -title: Monitoring V2 Configuration Examples +title: Monitoring Configuration Examples --- diff --git a/versioned_docs/version-2.6/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md b/versioned_docs/version-2.6/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md index 7fea6f9a7d9..27b0c7ff364 100644 --- a/versioned_docs/version-2.6/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md +++ b/versioned_docs/version-2.6/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md @@ -1,5 +1,5 @@ --- -title: Monitoring V2 Configuration +title: Monitoring Configuration Reference --- diff --git a/versioned_docs/version-2.6/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md b/versioned_docs/version-2.6/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md index 0e01bd3d293..1a0f0f0e5f8 100644 --- a/versioned_docs/version-2.6/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md +++ b/versioned_docs/version-2.6/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Rancher Architecture --- diff --git a/versioned_docs/version-2.6/reference-guides/rancher-security/rancher-security.md b/versioned_docs/version-2.6/reference-guides/rancher-security/rancher-security.md index 8b28ef1b3cb..9c982209fc5 100644 --- a/versioned_docs/version-2.6/reference-guides/rancher-security/rancher-security.md +++ b/versioned_docs/version-2.6/reference-guides/rancher-security/rancher-security.md @@ -1,5 +1,5 @@ --- -title: Security +title: Rancher Security Guides --- diff --git a/versioned_docs/version-2.7/faq/dockershim.md b/versioned_docs/version-2.7/faq/dockershim.md index 23b982f7b30..4e710c9746b 100644 --- a/versioned_docs/version-2.7/faq/dockershim.md +++ b/versioned_docs/version-2.7/faq/dockershim.md @@ -1,5 +1,5 @@ --- -title: Dockershim +title: Dockershim FAQ --- diff --git a/versioned_docs/version-2.7/faq/security.md b/versioned_docs/version-2.7/faq/security.md index 684444e7303..46f774bf722 100644 --- a/versioned_docs/version-2.7/faq/security.md +++ b/versioned_docs/version-2.7/faq/security.md @@ -1,5 +1,5 @@ --- -title: Security +title: Security FAQ --- diff --git a/versioned_docs/version-2.7/faq/technical-items.md b/versioned_docs/version-2.7/faq/technical-items.md index db2500b7fbf..42bfe966726 100644 --- a/versioned_docs/version-2.7/faq/technical-items.md +++ b/versioned_docs/version-2.7/faq/technical-items.md @@ -1,5 +1,5 @@ --- -title: Technical +title: Technical FAQ --- diff --git a/versioned_docs/version-2.7/faq/telemetry.md b/versioned_docs/version-2.7/faq/telemetry.md index a510347e556..edfcaebed4e 100644 --- a/versioned_docs/version-2.7/faq/telemetry.md +++ b/versioned_docs/version-2.7/faq/telemetry.md @@ -1,5 +1,5 @@ --- -title: Telemetry +title: Telemetry FAQ --- diff --git a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md index cbdeee0602b..cb01f789246 100644 --- a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md +++ b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md @@ -1,5 +1,5 @@ --- -title: Setup Guide +title: Istio Setup Guides --- diff --git a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md index 9fa587aef2b..e3de2cf69cf 100644 --- a/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md +++ b/versioned_docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md @@ -1,5 +1,5 @@ --- -title: Configuration +title: Monitoring Configuration Guides --- diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md index 228ec4c0992..e67a2da5d6d 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md @@ -1,5 +1,5 @@ --- -title: RKE Templates +title: About RKE1 Templates --- diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md index 6221b9b9e14..9331ca65d7a 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md @@ -1,5 +1,5 @@ --- -title: Backups and Disaster Recovery +title: Backup, Restore, and Disaster Recovery keywords: [rancher backup restore, rancher backup and restore, backup restore rancher, rancher backup and restore rancher] --- diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md index 7a79e5f5622..e361e6c2f02 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md @@ -1,12 +1,14 @@ --- -title: Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials. +title: Infrastructure Setup --- -To set up infrastructure for a high-availability K3s Kubernetes cluster with an external DB, refer to [this page.](ha-k3s-kubernetes-cluster.md) +Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials. + +To set up infrastructure for a high-availability K3s Kubernetes cluster with an external database, refer to [this page.](ha-k3s-kubernetes-cluster.md) To set up infrastructure for a high-availability RKE Kubernetes cluster, refer to [this page.](ha-rke1-kubernetes-cluster.md) diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md index 00c22ead1c3..05267457f70 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md @@ -1,11 +1,13 @@ --- -title: "Don't have a Kubernetes cluster? Try one of these tutorials." +title: Setting up a Kubernetes Cluster for Rancher Server --- +Don't have a Kubernetes cluster? Try one of these tutorials. + This section contains information on how to install a Kubernetes cluster that the Rancher server can be installed on. Rancher can run on any Kubernetes cluster. diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md index 230a3d6212b..fcacb555608 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md @@ -1,5 +1,5 @@ --- -title: Setting up Kubernetes Clusters in Rancher +title: Kubernetes Clusters in Rancher Setup description: Provisioning Kubernetes Clusters --- diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md index 8d084cb00d7..bacdd796fce 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md @@ -1,5 +1,5 @@ --- -title: The Horizontal Pod Autoscaler +title: Horizontal Pod Autoscaler description: Learn about the horizontal pod autoscaler (HPA). How to manage HPAs and how to test them with a service deployment --- diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md index 7879660225d..3e97882659c 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md @@ -1,5 +1,5 @@ --- -title: Kubernetes Resources +title: Kubernetes Resources Setup --- diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md index 934a552f019..48d3f52e276 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md @@ -1,5 +1,5 @@ --- -title: Set Up Load Balancer and Ingress Controller within Rancher +title: Load Balancer and Ingress Controller Setup within Rancher description: Learn how you can set up load balancers and ingress controllers to redirect service requests within Rancher, and learn about the limitations of load balancers --- diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md index ca181ffc07a..66a81355f0e 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md @@ -1,5 +1,5 @@ --- -title: Cluster Access +title: Access Clusters --- diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md index b7f9af5c766..a3d842f5265 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -1,5 +1,5 @@ --- -title: "Kubernetes Persistent Storage: Volumes and Storage Classes" +title: Create Kubernetes Persistent Volumes and Storage Classes description: "Learn about the two ways with which you can create persistent storage in Kubernetes: persistent volumes and storage classes" --- diff --git a/versioned_docs/version-2.7/integrations-in-rancher/fleet-gitops-at-scale/architecture.md b/versioned_docs/version-2.7/integrations-in-rancher/fleet-gitops-at-scale/architecture.md index 9d64e38de41..169b667a27e 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/fleet-gitops-at-scale/architecture.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/fleet-gitops-at-scale/architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Fleet Architecture --- diff --git a/versioned_docs/version-2.7/integrations-in-rancher/logging/logging-architecture.md b/versioned_docs/version-2.7/integrations-in-rancher/logging/logging-architecture.md index 958bc5d3069..f4b716a6c2e 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/logging/logging-architecture.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/logging/logging-architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Logging Architecture --- diff --git a/versioned_docs/version-2.7/integrations-in-rancher/longhorn.md b/versioned_docs/version-2.7/integrations-in-rancher/longhorn.md index d9daf5421a5..42f170c2852 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/longhorn.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/longhorn.md @@ -1,5 +1,5 @@ --- -title: Longhorn - Cloud native distributed block storage for Kubernetes +title: Cloud Native Storage with Longhorn --- diff --git a/versioned_docs/version-2.7/reference-guides/best-practices/best-practices.md b/versioned_docs/version-2.7/reference-guides/best-practices/best-practices.md index d2520a3e785..c0b4a9cc233 100644 --- a/versioned_docs/version-2.7/reference-guides/best-practices/best-practices.md +++ b/versioned_docs/version-2.7/reference-guides/best-practices/best-practices.md @@ -1,5 +1,5 @@ --- -title: Best Practices Guide +title: Best Practice Guides --- diff --git a/versioned_docs/version-2.7/reference-guides/monitoring-v2-configuration/examples.md b/versioned_docs/version-2.7/reference-guides/monitoring-v2-configuration/examples.md index d6ef1920256..1660d70a31f 100644 --- a/versioned_docs/version-2.7/reference-guides/monitoring-v2-configuration/examples.md +++ b/versioned_docs/version-2.7/reference-guides/monitoring-v2-configuration/examples.md @@ -1,5 +1,5 @@ --- -title: Monitoring V2 Configuration Examples +title: Monitoring Configuration Examples --- diff --git a/versioned_docs/version-2.7/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md b/versioned_docs/version-2.7/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md index 7fea6f9a7d9..27b0c7ff364 100644 --- a/versioned_docs/version-2.7/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md +++ b/versioned_docs/version-2.7/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md @@ -1,5 +1,5 @@ --- -title: Monitoring V2 Configuration +title: Monitoring Configuration Reference --- diff --git a/versioned_docs/version-2.7/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md b/versioned_docs/version-2.7/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md index 0e01bd3d293..1a0f0f0e5f8 100644 --- a/versioned_docs/version-2.7/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md +++ b/versioned_docs/version-2.7/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Rancher Architecture --- diff --git a/versioned_docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md b/versioned_docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md index 88f00624b20..90f9b494857 100644 --- a/versioned_docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md +++ b/versioned_docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md @@ -1,5 +1,5 @@ --- -title: K3s Hardening Guide +title: K3s Hardening Guides --- diff --git a/versioned_docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md b/versioned_docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md index c7992611f8a..22c465eb07d 100644 --- a/versioned_docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md +++ b/versioned_docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md @@ -1,5 +1,5 @@ --- -title: RKE Hardening Guide +title: RKE Hardening Guides --- diff --git a/versioned_docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md b/versioned_docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md index a69c97665bb..bf79c65bd58 100644 --- a/versioned_docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md +++ b/versioned_docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md @@ -1,5 +1,5 @@ --- -title: RKE2 Hardening Guide +title: RKE2 Hardening Guides --- diff --git a/versioned_docs/version-2.7/reference-guides/rancher-security/rancher-security.md b/versioned_docs/version-2.7/reference-guides/rancher-security/rancher-security.md index 70f92fa8d77..1d7e12f113d 100644 --- a/versioned_docs/version-2.7/reference-guides/rancher-security/rancher-security.md +++ b/versioned_docs/version-2.7/reference-guides/rancher-security/rancher-security.md @@ -1,5 +1,6 @@ --- -title: Security +title: Rancher Security Guides + --- diff --git a/versioned_docs/version-2.8/faq/dockershim.md b/versioned_docs/version-2.8/faq/dockershim.md index 23b982f7b30..4e710c9746b 100644 --- a/versioned_docs/version-2.8/faq/dockershim.md +++ b/versioned_docs/version-2.8/faq/dockershim.md @@ -1,5 +1,5 @@ --- -title: Dockershim +title: Dockershim FAQ --- diff --git a/versioned_docs/version-2.8/faq/security.md b/versioned_docs/version-2.8/faq/security.md index 684444e7303..46f774bf722 100644 --- a/versioned_docs/version-2.8/faq/security.md +++ b/versioned_docs/version-2.8/faq/security.md @@ -1,5 +1,5 @@ --- -title: Security +title: Security FAQ --- diff --git a/versioned_docs/version-2.8/faq/technical-items.md b/versioned_docs/version-2.8/faq/technical-items.md index db2500b7fbf..42bfe966726 100644 --- a/versioned_docs/version-2.8/faq/technical-items.md +++ b/versioned_docs/version-2.8/faq/technical-items.md @@ -1,5 +1,5 @@ --- -title: Technical +title: Technical FAQ --- diff --git a/versioned_docs/version-2.8/faq/telemetry.md b/versioned_docs/version-2.8/faq/telemetry.md index a510347e556..edfcaebed4e 100644 --- a/versioned_docs/version-2.8/faq/telemetry.md +++ b/versioned_docs/version-2.8/faq/telemetry.md @@ -1,5 +1,5 @@ --- -title: Telemetry +title: Telemetry FAQ --- diff --git a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md index cbdeee0602b..cb01f789246 100644 --- a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md +++ b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md @@ -1,5 +1,5 @@ --- -title: Setup Guide +title: Istio Setup Guides --- diff --git a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md index 9fa587aef2b..e3de2cf69cf 100644 --- a/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md +++ b/versioned_docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md @@ -1,5 +1,5 @@ --- -title: Configuration +title: Monitoring Configuration Guides --- diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md index 228ec4c0992..e67a2da5d6d 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md @@ -1,5 +1,5 @@ --- -title: RKE Templates +title: About RKE1 Templates --- diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md index e6ebbf13c5b..c202d71be45 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md @@ -1,5 +1,5 @@ --- -title: Backups and Disaster Recovery +title: Backup, Restore, and Disaster Recovery keywords: [rancher backup restore, rancher backup and restore, backup restore rancher, rancher backup and restore rancher] --- diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md index 7a79e5f5622..e361e6c2f02 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md @@ -1,12 +1,14 @@ --- -title: Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials. +title: Infrastructure Setup --- -To set up infrastructure for a high-availability K3s Kubernetes cluster with an external DB, refer to [this page.](ha-k3s-kubernetes-cluster.md) +Don't have infrastructure for your Kubernetes cluster? Try one of these tutorials. + +To set up infrastructure for a high-availability K3s Kubernetes cluster with an external database, refer to [this page.](ha-k3s-kubernetes-cluster.md) To set up infrastructure for a high-availability RKE Kubernetes cluster, refer to [this page.](ha-rke1-kubernetes-cluster.md) diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md index 00c22ead1c3..05267457f70 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md @@ -1,11 +1,13 @@ --- -title: "Don't have a Kubernetes cluster? Try one of these tutorials." +title: Setting up a Kubernetes Cluster for Rancher Server --- +Don't have a Kubernetes cluster? Try one of these tutorials. + This section contains information on how to install a Kubernetes cluster that the Rancher server can be installed on. Rancher can run on any Kubernetes cluster. diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md index 230a3d6212b..fcacb555608 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md @@ -1,5 +1,5 @@ --- -title: Setting up Kubernetes Clusters in Rancher +title: Kubernetes Clusters in Rancher Setup description: Provisioning Kubernetes Clusters --- diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md index 8d084cb00d7..bacdd796fce 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md @@ -1,5 +1,5 @@ --- -title: The Horizontal Pod Autoscaler +title: Horizontal Pod Autoscaler description: Learn about the horizontal pod autoscaler (HPA). How to manage HPAs and how to test them with a service deployment --- diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md index 7879660225d..3e97882659c 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md @@ -1,5 +1,5 @@ --- -title: Kubernetes Resources +title: Kubernetes Resources Setup --- diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md index 934a552f019..48d3f52e276 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md @@ -1,5 +1,5 @@ --- -title: Set Up Load Balancer and Ingress Controller within Rancher +title: Load Balancer and Ingress Controller Setup within Rancher description: Learn how you can set up load balancers and ingress controllers to redirect service requests within Rancher, and learn about the limitations of load balancers --- diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md index ca181ffc07a..66a81355f0e 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md @@ -1,5 +1,5 @@ --- -title: Cluster Access +title: Access Clusters --- diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md index 8b2d0db56c7..7ada9cd85af 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -1,5 +1,5 @@ --- -title: "Kubernetes Persistent Storage: Volumes and Storage Classes" +title: Create Kubernetes Persistent Volumes and Storage Classes description: "Learn about the two ways with which you can create persistent storage in Kubernetes: persistent volumes and storage classes" --- diff --git a/versioned_docs/version-2.8/integrations-in-rancher/fleet/architecture.md b/versioned_docs/version-2.8/integrations-in-rancher/fleet/architecture.md index dcdd07ce36f..556bd33f4e8 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/fleet/architecture.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/fleet/architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Fleet Architecture --- diff --git a/versioned_docs/version-2.8/integrations-in-rancher/logging/logging-architecture.md b/versioned_docs/version-2.8/integrations-in-rancher/logging/logging-architecture.md index 958bc5d3069..f4b716a6c2e 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/logging/logging-architecture.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/logging/logging-architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Logging Architecture --- diff --git a/versioned_docs/version-2.8/reference-guides/best-practices/best-practices.md b/versioned_docs/version-2.8/reference-guides/best-practices/best-practices.md index d2520a3e785..c0b4a9cc233 100644 --- a/versioned_docs/version-2.8/reference-guides/best-practices/best-practices.md +++ b/versioned_docs/version-2.8/reference-guides/best-practices/best-practices.md @@ -1,5 +1,5 @@ --- -title: Best Practices Guide +title: Best Practice Guides --- diff --git a/versioned_docs/version-2.8/reference-guides/monitoring-v2-configuration/examples.md b/versioned_docs/version-2.8/reference-guides/monitoring-v2-configuration/examples.md index d6ef1920256..1660d70a31f 100644 --- a/versioned_docs/version-2.8/reference-guides/monitoring-v2-configuration/examples.md +++ b/versioned_docs/version-2.8/reference-guides/monitoring-v2-configuration/examples.md @@ -1,5 +1,5 @@ --- -title: Monitoring V2 Configuration Examples +title: Monitoring Configuration Examples --- diff --git a/versioned_docs/version-2.8/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md b/versioned_docs/version-2.8/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md index 7fea6f9a7d9..27b0c7ff364 100644 --- a/versioned_docs/version-2.8/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md +++ b/versioned_docs/version-2.8/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md @@ -1,5 +1,5 @@ --- -title: Monitoring V2 Configuration +title: Monitoring Configuration Reference --- diff --git a/versioned_docs/version-2.8/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md b/versioned_docs/version-2.8/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md index 0e01bd3d293..1a0f0f0e5f8 100644 --- a/versioned_docs/version-2.8/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md +++ b/versioned_docs/version-2.8/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md @@ -1,5 +1,5 @@ --- -title: Architecture +title: Rancher Architecture --- diff --git a/versioned_docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md b/versioned_docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md index 88f00624b20..90f9b494857 100644 --- a/versioned_docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md +++ b/versioned_docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md @@ -1,5 +1,5 @@ --- -title: K3s Hardening Guide +title: K3s Hardening Guides --- diff --git a/versioned_docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md b/versioned_docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md index cc91f95c044..138c9439271 100644 --- a/versioned_docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md +++ b/versioned_docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md @@ -1,5 +1,5 @@ --- -title: RKE Hardening Guide +title: RKE Hardening Guides --- diff --git a/versioned_docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md b/versioned_docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md index a69c97665bb..bf79c65bd58 100644 --- a/versioned_docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md +++ b/versioned_docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md @@ -1,5 +1,5 @@ --- -title: RKE2 Hardening Guide +title: RKE2 Hardening Guides --- diff --git a/versioned_docs/version-2.8/reference-guides/rancher-security/rancher-security.md b/versioned_docs/version-2.8/reference-guides/rancher-security/rancher-security.md index 02d9b400b21..795fed1d87e 100644 --- a/versioned_docs/version-2.8/reference-guides/rancher-security/rancher-security.md +++ b/versioned_docs/version-2.8/reference-guides/rancher-security/rancher-security.md @@ -1,5 +1,5 @@ --- -title: Security +title: Rancher Security Guides --- diff --git a/versioned_sidebars/version-2.0-2.4-sidebars.json b/versioned_sidebars/version-2.0-2.4-sidebars.json index 4f8b7534f10..c3ad1072c3a 100644 --- a/versioned_sidebars/version-2.0-2.4-sidebars.json +++ b/versioned_sidebars/version-2.0-2.4-sidebars.json @@ -32,7 +32,7 @@ "getting-started/quick-start-guides/cli", { "type": "category", - "label": "Deploying Rancher", + "label": "Deploying Rancher Server", "link": { "type": "doc", "id": "getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager" @@ -375,7 +375,7 @@ "items": [ { "type": "category", - "label": "Kubernetes Cluster Setup", + "label": "Setting up a Kubernetes Cluster for Rancher Server", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup" @@ -459,7 +459,7 @@ "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster", { "type": "category", - "label": "vSphere", + "label": "Creating a vSphere Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere" @@ -855,7 +855,7 @@ }, { "type": "category", - "label": "Istio Setup Guide", + "label": "Istio Setup Guides", "link": { "type": "doc", "id": "how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide" @@ -901,7 +901,7 @@ "items": [ { "type": "category", - "label": "Best Practices", + "label": "Best Practice Guides", "link": { "type": "doc", "id": "reference-guides/best-practices/best-practices" @@ -1104,7 +1104,7 @@ }, { "type": "category", - "label": "Rancher Security", + "label": "Rancher Security Guides", "link": { "type": "doc", "id": "reference-guides/rancher-security/rancher-security" diff --git a/versioned_sidebars/version-2.5-sidebars.json b/versioned_sidebars/version-2.5-sidebars.json index f924143110e..b9ac6a84bce 100644 --- a/versioned_sidebars/version-2.5-sidebars.json +++ b/versioned_sidebars/version-2.5-sidebars.json @@ -31,7 +31,7 @@ "items": [ { "type": "category", - "label": "Deploying Rancher", + "label": "Deploying Rancher Server", "link": { "type": "doc", "id": "getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager" @@ -234,7 +234,7 @@ "items": [ { "type": "category", - "label": "Kubernetes Cluster Setup", + "label": "Setting up a Kubernetes Cluster for Rancher Server", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup" @@ -320,7 +320,7 @@ "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster", { "type": "category", - "label": "vSphere", + "label": "Creating a vSphere Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere" @@ -368,7 +368,7 @@ }, { "type": "category", - "label": "vSphere", + "label": "Creating a vSphere Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/set-up-cloud-providers/vsphere/vsphere" @@ -750,7 +750,7 @@ }, { "type": "category", - "label": "Istio Setup Guide", + "label": "Istio Setup Guides", "link": { "type": "doc", "id": "how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide" @@ -797,7 +797,7 @@ "items": [ { "type": "category", - "label": "Best Practices", + "label": "Best Practice Guides", "link": { "type": "doc", "id": "reference-guides/best-practices/best-practices" @@ -865,7 +865,7 @@ "reference-guides/cluster-configuration/rancher-server-configuration/eks-cluster-configuration", { "type": "category", - "label": "GKE Cluster Configuration", + "label": "GKE Cluster Configuration Reference", "link": { "type": "doc", "id": "reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration" @@ -1035,7 +1035,7 @@ "reference-guides/rke1-template-example-yaml", { "type": "category", - "label": "Rancher Security", + "label": "Rancher Security Guides", "link": { "type": "doc", "id": "reference-guides/rancher-security/rancher-security" diff --git a/versioned_sidebars/version-2.6-sidebars.json b/versioned_sidebars/version-2.6-sidebars.json index 71c34c86d52..8e52facf872 100644 --- a/versioned_sidebars/version-2.6-sidebars.json +++ b/versioned_sidebars/version-2.6-sidebars.json @@ -16,7 +16,7 @@ "items": [ { "type": "category", - "label": "Deploying Rancher", + "label": "Deploying Rancher Server", "link": { "type": "doc", "id": "getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager" @@ -372,7 +372,7 @@ }, { "type": "category", - "label": "Kubernetes Cluster Setup", + "label": "Setting up a Kubernetes Cluster for Rancher Server", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup" @@ -492,7 +492,7 @@ "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster", { "type": "category", - "label": "vSphere", + "label": "Creating a vSphere Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere" @@ -505,7 +505,7 @@ }, { "type": "category", - "label": "Nutanix", + "label": "Creating a Nutanix AOS Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix" @@ -689,7 +689,7 @@ }, { "type": "category", - "label": "Monitoring V2 Configuration Guides", + "label": "Monitoring Configuration Guides", "link": { "type": "doc", "id": "how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides" @@ -712,7 +712,7 @@ }, { "type": "category", - "label": "Istio Setup Guide", + "label": "Istio Setup Guides", "link": { "type": "doc", "id": "how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide" @@ -773,7 +773,7 @@ "items": [ { "type": "category", - "label": "Best Practices", + "label": "Best Practice Guides", "link": { "type": "doc", "id": "reference-guides/best-practices/best-practices" @@ -845,7 +845,7 @@ "reference-guides/cluster-configuration/rancher-server-configuration/aks-cluster-configuration", { "type": "category", - "label": "GKE Cluster Configuration", + "label": "GKE Cluster Configuration Reference", "link": { "type": "doc", "id": "reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration" @@ -937,7 +937,7 @@ "reference-guides/kubernetes-concepts", { "type": "category", - "label": "Monitoring V2 Configuration", + "label": "Monitoring Configuration Reference", "link": { "type": "doc", "id": "reference-guides/monitoring-v2-configuration/monitoring-v2-configuration" @@ -1019,7 +1019,7 @@ }, { "type": "category", - "label": "Rancher Security", + "label": "Rancher Security Guides", "link": { "type": "doc", "id": "reference-guides/rancher-security/rancher-security" diff --git a/versioned_sidebars/version-2.7-sidebars.json b/versioned_sidebars/version-2.7-sidebars.json index 69a36db52da..11fc615225d 100644 --- a/versioned_sidebars/version-2.7-sidebars.json +++ b/versioned_sidebars/version-2.7-sidebars.json @@ -16,7 +16,7 @@ "items": [ { "type": "category", - "label": "Deploying Rancher", + "label": "Deploying Rancher Server", "link": { "type": "doc", "id": "getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager" @@ -376,7 +376,7 @@ }, { "type": "category", - "label": "Kubernetes Cluster Setup", + "label": "Setting up a Kubernetes Cluster for Rancher Server", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup" @@ -503,7 +503,7 @@ "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster", { "type": "category", - "label": "vSphere", + "label": "Creating a vSphere Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere" @@ -516,7 +516,7 @@ }, { "type": "category", - "label": "Nutanix", + "label": "Creating a Nutanix AOS Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix" @@ -686,7 +686,7 @@ }, { "type": "category", - "label": "Monitoring V2 Configuration Guides", + "label": "Monitoring Configuration Guides", "link": { "type": "doc", "id": "how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides" @@ -709,7 +709,7 @@ }, { "type": "category", - "label": "Istio Setup Guide", + "label": "Istio Setup Guides", "link": { "type": "doc", "id": "how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide" @@ -771,7 +771,7 @@ "items": [ { "type": "category", - "label": "Best Practices", + "label": "Best Practice Guides", "link": { "type": "doc", "id": "reference-guides/best-practices/best-practices" @@ -843,7 +843,7 @@ "reference-guides/cluster-configuration/rancher-server-configuration/aks-cluster-configuration", { "type": "category", - "label": "GKE Cluster Configuration", + "label": "GKE Cluster Configuration Reference", "link": { "type": "doc", "id": "reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration" @@ -935,7 +935,7 @@ "reference-guides/kubernetes-concepts", { "type": "category", - "label": "Monitoring V2 Configuration", + "label": "Monitoring Configuration Reference", "link": { "type": "doc", "id": "reference-guides/monitoring-v2-configuration/monitoring-v2-configuration" @@ -1003,7 +1003,7 @@ "reference-guides/rancher-webhook", { "type": "category", - "label": "Rancher Security", + "label": "Rancher Security Guides", "link": { "type": "doc", "id": "reference-guides/rancher-security/rancher-security" diff --git a/versioned_sidebars/version-2.8-sidebars.json b/versioned_sidebars/version-2.8-sidebars.json index 7345f82e749..de5f50aa6ec 100644 --- a/versioned_sidebars/version-2.8-sidebars.json +++ b/versioned_sidebars/version-2.8-sidebars.json @@ -16,7 +16,7 @@ "items": [ { "type": "category", - "label": "Deploying Rancher", + "label": "Deploying Rancher Server", "link": { "type": "doc", "id": "getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager" @@ -376,7 +376,7 @@ }, { "type": "category", - "label": "Kubernetes Cluster Setup", + "label": "Setting up a Kubernetes Cluster for Rancher Server", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup" @@ -503,7 +503,7 @@ "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster", { "type": "category", - "label": "vSphere", + "label": "Creating a vSphere Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere" @@ -516,7 +516,7 @@ }, { "type": "category", - "label": "Nutanix", + "label": "Creating a Nutanix AOS Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix" @@ -686,7 +686,7 @@ }, { "type": "category", - "label": "Monitoring V2 Configuration Guides", + "label": "Monitoring Configuration Guides", "link": { "type": "doc", "id": "how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides" @@ -709,7 +709,7 @@ }, { "type": "category", - "label": "Istio Setup Guide", + "label": "Istio Setup Guides", "link": { "type": "doc", "id": "how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide" @@ -771,7 +771,7 @@ "items": [ { "type": "category", - "label": "Best Practices", + "label": "Best Practice Guides", "link": { "type": "doc", "id": "reference-guides/best-practices/best-practices" @@ -843,7 +843,7 @@ "reference-guides/cluster-configuration/rancher-server-configuration/aks-cluster-configuration", { "type": "category", - "label": "GKE Cluster Configuration", + "label": "GKE Cluster Configuration Reference", "link": { "type": "doc", "id": "reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration" @@ -935,7 +935,7 @@ "reference-guides/kubernetes-concepts", { "type": "category", - "label": "Monitoring V2 Configuration", + "label": "Monitoring Configuration Reference", "link": { "type": "doc", "id": "reference-guides/monitoring-v2-configuration/monitoring-v2-configuration" @@ -1003,7 +1003,7 @@ "reference-guides/rancher-webhook", { "type": "category", - "label": "Rancher Security", + "label": "Rancher Security Guides", "link": { "type": "doc", "id": "reference-guides/rancher-security/rancher-security" From eda6da5c7f3c3534b1097dc68c893da34480c94f Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Tue, 9 Apr 2024 17:11:32 -0400 Subject: [PATCH 35/67] #911 migration doc in need of clarification (#1193) * 911 Migration doc in need of clarification Clarified value * dns step added * revise wording * revised based on 760 * Apply suggestions from code review Co-authored-by: Billy Tat * > * rm double newlines --------- Co-authored-by: Billy Tat --- .../migrate-rancher-to-new-cluster.md | 14 +++++++++----- .../migrate-rancher-to-new-cluster.md | 12 +++++++++--- .../migrate-rancher-to-new-cluster.md | 12 ++++++++---- .../migrate-rancher-to-new-cluster.md | 12 ++++++++---- 4 files changed, 34 insertions(+), 16 deletions(-) diff --git a/docs/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md b/docs/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md index 89faf8d11bf..676d09f94ed 100644 --- a/docs/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md +++ b/docs/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md @@ -8,14 +8,13 @@ title: Migrating Rancher to a New Cluster If you are migrating Rancher to a new Kubernetes cluster, you don't need to install Rancher on the new cluster first. If Rancher is restored to a new cluster with Rancher already installed, it can cause problems. - ### Prerequisites These instructions assume that you have [created a backup](back-up-rancher.md) and already installed a new Kubernetes cluster where Rancher will be deployed. The backup is specific to the Rancher application and can only migrate the Rancher application. :::caution -It is required to use the same hostname that was set as the server URL in the first cluster. If not done, downstream clusters will show as unavailable in the cluster management page of the UI, and you won't be able to click inside the cluster or on the cluster's Explore button. +You must use the same hostname that was set as the server URL in the original cluster. If you don't, downstream clusters will show as unavailable in the cluster management page of the UI, and you won't be able to click inside the cluster or on the cluster's **Explore** button. ::: @@ -25,7 +24,6 @@ Rancher can be installed on any Kubernetes cluster, including hosted Kubernetes Since Rancher can be installed on any Kubernetes cluster, you can use this backup and restore method to migrate Rancher from one Kubernetes cluster to any other Kubernetes cluster. This method *only* migrates Rancher-related resources and won't affect other applications on the cluster. Refer to the [support matrix](https://www.suse.com/lifecycle/) to identify which Kubernetes cluster types and versions are supported for your Rancher version. - ### 1. Install the rancher-backup Helm chart Install the [`rancher-backup chart`](https://github.com/rancher/backup-restore-operator/tags): @@ -54,10 +52,10 @@ Install the [`rancher-backup chart`](https://github.com/rancher/backup-restore-o The above assumes an environment with outbound connectivity to Docker Hub. - For an **air-gapped environment**, use the Helm value below to pull the `backup-restore-operator` image from your private registry when installing the rancher-backup Helm chart. + For an **air-gapped environment**, use the following Helm value to pull the `backup-restore-operator` image from your private registry when you install the rancher-backup Helm chart. ```bash - --set image.repository $REGISTRY/rancher/backup-restore-operator + --set image.repository /rancher/backup-restore-operator ``` ::: @@ -189,3 +187,9 @@ helm install rancher rancher-latest/rancher -n cattle-system -f rancher-values.y ``` ::: + +### 5. Redirect Traffic to the New Cluster + +After migration completes, update your DNS records and any load balancers, so that traffic is routed correctly to the migrated cluster. Remember that you must use the same hostname that was set as the server URL in the original cluster. + +Full instructions on how to redirect traffic to the migrated cluster differ based on your specific environment. Refer to your hosting provider's documentation for more details. diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md index 346c54d50c7..7df2e8dd9c0 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md @@ -52,10 +52,10 @@ Install the [`rancher-backup chart`](https://github.com/rancher/backup-restore-o The above assumes an environment with outbound connectivity to Docker Hub. - For an **air-gapped environment**, use the Helm value below to pull the `backup-restore-operator` image from your private registry when installing the rancher-backup Helm chart. + For an **air-gapped environment**, use the following Helm value to pull the `backup-restore-operator` image from your private registry when you install the rancher-backup Helm chart. ```bash - --set image.repository $REGISTRY/rancher/backup-restore-operator + --set image.repository /rancher/backup-restore-operator ``` ::: @@ -184,4 +184,10 @@ These values can be reused using the `rancher-values.yaml` file. Be sure to swit helm install rancher rancher-latest/rancher -n cattle-system -f rancher-values.yaml --version x.y.z ``` -::: \ No newline at end of file +::: + +### 5. Redirect Traffic to the New Cluster + +After migration completes, update your DNS records and any load balancers, so that traffic is routed correctly to the migrated cluster. Remember that you must use the same hostname that was set as the server URL in the original cluster. + +Full instructions on how to redirect traffic to the migrated cluster differ based on your specific environment. Refer to your hosting provider's documentation for more details. diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md index 89faf8d11bf..fc10b2f5d47 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md @@ -8,7 +8,6 @@ title: Migrating Rancher to a New Cluster If you are migrating Rancher to a new Kubernetes cluster, you don't need to install Rancher on the new cluster first. If Rancher is restored to a new cluster with Rancher already installed, it can cause problems. - ### Prerequisites These instructions assume that you have [created a backup](back-up-rancher.md) and already installed a new Kubernetes cluster where Rancher will be deployed. The backup is specific to the Rancher application and can only migrate the Rancher application. @@ -25,7 +24,6 @@ Rancher can be installed on any Kubernetes cluster, including hosted Kubernetes Since Rancher can be installed on any Kubernetes cluster, you can use this backup and restore method to migrate Rancher from one Kubernetes cluster to any other Kubernetes cluster. This method *only* migrates Rancher-related resources and won't affect other applications on the cluster. Refer to the [support matrix](https://www.suse.com/lifecycle/) to identify which Kubernetes cluster types and versions are supported for your Rancher version. - ### 1. Install the rancher-backup Helm chart Install the [`rancher-backup chart`](https://github.com/rancher/backup-restore-operator/tags): @@ -54,10 +52,10 @@ Install the [`rancher-backup chart`](https://github.com/rancher/backup-restore-o The above assumes an environment with outbound connectivity to Docker Hub. - For an **air-gapped environment**, use the Helm value below to pull the `backup-restore-operator` image from your private registry when installing the rancher-backup Helm chart. + For an **air-gapped environment**, use the following Helm value to pull the `backup-restore-operator` image from your private registry when you install the rancher-backup Helm chart. ```bash - --set image.repository $REGISTRY/rancher/backup-restore-operator + --set image.repository /rancher/backup-restore-operator ``` ::: @@ -189,3 +187,9 @@ helm install rancher rancher-latest/rancher -n cattle-system -f rancher-values.y ``` ::: + +### 5. Redirect Traffic to the New Cluster + +After migration completes, update your DNS records and any load balancers, so that traffic is routed correctly to the migrated cluster. Remember that you must use the same hostname that was set as the server URL in the original cluster. + +Full instructions on how to redirect traffic to the migrated cluster differ based on your specific environment. Refer to your hosting provider's documentation for more details. diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md index 89faf8d11bf..fc10b2f5d47 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md @@ -8,7 +8,6 @@ title: Migrating Rancher to a New Cluster If you are migrating Rancher to a new Kubernetes cluster, you don't need to install Rancher on the new cluster first. If Rancher is restored to a new cluster with Rancher already installed, it can cause problems. - ### Prerequisites These instructions assume that you have [created a backup](back-up-rancher.md) and already installed a new Kubernetes cluster where Rancher will be deployed. The backup is specific to the Rancher application and can only migrate the Rancher application. @@ -25,7 +24,6 @@ Rancher can be installed on any Kubernetes cluster, including hosted Kubernetes Since Rancher can be installed on any Kubernetes cluster, you can use this backup and restore method to migrate Rancher from one Kubernetes cluster to any other Kubernetes cluster. This method *only* migrates Rancher-related resources and won't affect other applications on the cluster. Refer to the [support matrix](https://www.suse.com/lifecycle/) to identify which Kubernetes cluster types and versions are supported for your Rancher version. - ### 1. Install the rancher-backup Helm chart Install the [`rancher-backup chart`](https://github.com/rancher/backup-restore-operator/tags): @@ -54,10 +52,10 @@ Install the [`rancher-backup chart`](https://github.com/rancher/backup-restore-o The above assumes an environment with outbound connectivity to Docker Hub. - For an **air-gapped environment**, use the Helm value below to pull the `backup-restore-operator` image from your private registry when installing the rancher-backup Helm chart. + For an **air-gapped environment**, use the following Helm value to pull the `backup-restore-operator` image from your private registry when you install the rancher-backup Helm chart. ```bash - --set image.repository $REGISTRY/rancher/backup-restore-operator + --set image.repository /rancher/backup-restore-operator ``` ::: @@ -189,3 +187,9 @@ helm install rancher rancher-latest/rancher -n cattle-system -f rancher-values.y ``` ::: + +### 5. Redirect Traffic to the New Cluster + +After migration completes, update your DNS records and any load balancers, so that traffic is routed correctly to the migrated cluster. Remember that you must use the same hostname that was set as the server URL in the original cluster. + +Full instructions on how to redirect traffic to the migrated cluster differ based on your specific environment. Refer to your hosting provider's documentation for more details. From 14ed8e0dc34de12d6000bddd96c91da1dadc5240 Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Wed, 10 Apr 2024 07:05:42 -0700 Subject: [PATCH 36/67] [skip ci] Update Algolia config reference (#1227) --- dev-resources/algolia-search/config.json | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/dev-resources/algolia-search/config.json b/dev-resources/algolia-search/config.json index db628e87078..4a39b31e415 100644 --- a/dev-resources/algolia-search/config.json +++ b/dev-resources/algolia-search/config.json @@ -8,7 +8,8 @@ "lvl2": "article h2", "lvl3": "article h3", "lvl4": "article h4", - "lvl5": "article h5" + "lvl5": "article h5", + "lvl6": "article h6" }, "custom_settings": { "attributesForFaceting": [ From 00ca7f6630d2609f4a82b2b5b7fbaa0160a7389e Mon Sep 17 00:00:00 2001 From: Alexandra Settle Date: Thu, 11 Apr 2024 00:35:25 +1000 Subject: [PATCH 37/67] Updating Prime text to improve clarity (#1225) * Updating Prime text to improve clarity * Apply suggestions from code review Co-authored-by: Marty Hernandez Avedon --------- Co-authored-by: Marty Hernandez Avedon --- .../quick-start-guides/deploy-rancher-manager/prime.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started/quick-start-guides/deploy-rancher-manager/prime.md b/docs/getting-started/quick-start-guides/deploy-rancher-manager/prime.md index 26700be8cf4..b1d4c281594 100644 --- a/docs/getting-started/quick-start-guides/deploy-rancher-manager/prime.md +++ b/docs/getting-started/quick-start-guides/deploy-rancher-manager/prime.md @@ -6,6 +6,6 @@ title: Rancher Prime -Rancher v2.7 introduces Rancher Prime, an evolution of the Rancher enterprise offering. Rancher Prime is a new edition of the commercial, enterprise offering built on the the same source code. Rancher’s product will therefore continue to be 100% open source with additional value coming in from security assurances, extended lifecycles, access to focused architectures and Kubernetes advisories. Rancher Prime will also offer options to get production support for innovative Rancher projects. With Rancher Prime, installation assets are hosted on a trusted registry owned and managed by Rancher. +SUSE Rancher introduces Rancher Prime – an evolution of Rancher – from version v2.7. Rancher Prime is the new commercially available enterprise offering of Rancher, built on the same open source code. The Rancher project will continue to be 100% open source. Prime introduces additional value with greater security assurances, extended lifecycles, access to focused architectures and Kubernetes advisories. Rancher Prime will also offer options to get production support for innovative Rancher projects. With Rancher Prime, installation assets are hosted on a trusted registry owned and managed by Rancher. To get started with Rancher Prime, [go to this page](https://www.rancher.com/quick-start) and fill out the form. From f76c9e5790e4fd0e5c0e99110bb2306277034a50 Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Wed, 10 Apr 2024 10:36:37 -0400 Subject: [PATCH 38/67] version sync for 1225 (#1229) --- .../quick-start-guides/deploy-rancher-manager/prime.md | 2 +- .../quick-start-guides/deploy-rancher-manager/prime.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/prime.md b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/prime.md index 26700be8cf4..b1d4c281594 100644 --- a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/prime.md +++ b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/prime.md @@ -6,6 +6,6 @@ title: Rancher Prime -Rancher v2.7 introduces Rancher Prime, an evolution of the Rancher enterprise offering. Rancher Prime is a new edition of the commercial, enterprise offering built on the the same source code. Rancher’s product will therefore continue to be 100% open source with additional value coming in from security assurances, extended lifecycles, access to focused architectures and Kubernetes advisories. Rancher Prime will also offer options to get production support for innovative Rancher projects. With Rancher Prime, installation assets are hosted on a trusted registry owned and managed by Rancher. +SUSE Rancher introduces Rancher Prime – an evolution of Rancher – from version v2.7. Rancher Prime is the new commercially available enterprise offering of Rancher, built on the same open source code. The Rancher project will continue to be 100% open source. Prime introduces additional value with greater security assurances, extended lifecycles, access to focused architectures and Kubernetes advisories. Rancher Prime will also offer options to get production support for innovative Rancher projects. With Rancher Prime, installation assets are hosted on a trusted registry owned and managed by Rancher. To get started with Rancher Prime, [go to this page](https://www.rancher.com/quick-start) and fill out the form. diff --git a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/prime.md b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/prime.md index 32759ea3865..e1b27128cb7 100644 --- a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/prime.md +++ b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/prime.md @@ -6,7 +6,7 @@ title: Rancher Prime -Prime is the Rancher ecosystem’s enterprise offering, with additional security, extended lifecycles, and access to Prime-exclusive documentation. Rancher Prime installation assets are hosted on a trusted SUSE registry, owned and managed by Rancher. The trusted Prime registry includes only stable releases that have been community-tested. +SUSE Rancher introduces Rancher Prime – an evolution of Rancher – from version v2.7. Rancher Prime is the new commercially available enterprise offering of Rancher, built on the same open source code. The Rancher project will continue to be 100% open source. Prime introduces additional value with greater security assurances, extended lifecycles, access to focused architectures and Kubernetes advisories. Rancher Prime will also offer options to get production support for innovative Rancher projects. With Rancher Prime, installation assets are hosted on a trusted registry owned and managed by Rancher. Prime also offers options for production support, as well as add-ons to your subscription that tailor to your commercial needs. From 14a4a1189d70d859ff7dc0dccfbabe980baaf3f8 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Wed, 10 Apr 2024 08:43:02 -0700 Subject: [PATCH 39/67] Updating 2.8.3 support matrix as page is live now. Signed-off-by: Sunil Singh --- src/pages/versions.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/pages/versions.md b/src/pages/versions.md index 7458e24a43d..ed4459c4c74 100644 --- a/src/pages/versions.md +++ b/src/pages/versions.md @@ -21,7 +21,7 @@ Here you can find links to supporting documentation for the current released ver v2.8.3 Documentation Release Notes -
N/A
+ Support Matrix
✓
✓
From b3a2a2ac486740904a2bc7db8a1f49101df5539f Mon Sep 17 00:00:00 2001 From: Jing Wu <17302487661@163.com> Date: Thu, 11 Apr 2024 04:49:38 +0800 Subject: [PATCH 40/67] Update missing Chinese translations of v2.8 and latest versions (#1220) * Update missing latest Chinese translations * Update missing v2.8 Chinese translations --- .../about-the-api/about-the-api.md | 93 + .../cli-with-rancher/cli-with-rancher.md | 9 + .../cli-with-rancher/rancher-cli.md | 4 + .../hardening-guides/hardening-guides.md | 55 + .../k3s-hardening-guide.md | 744 ++++ ...sessment-guide-with-cis-v1.23-k8s-v1.23.md | 34 +- ...sessment-guide-with-cis-v1.23-k8s-v1.24.md | 3148 ---------------- ...sessment-guide-with-cis-v1.23-k8s-v1.25.md | 3148 ---------------- ...sessment-guide-with-cis-v1.24-k8s-v1.24.md | 3208 ++++++++++++++++ ...ide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md | 3215 +++++++++++++++++ ...ssessment-guide-with-cis-v1.7-k8s-v1.25.md | 3148 ---------------- .../rke1-hardening-guide.md | 515 +++ ...sessment-guide-with-cis-v1.23-k8s-v1.23.md | 26 +- ...sessment-guide-with-cis-v1.23-k8s-v1.24.md | 3084 ---------------- ...sessment-guide-with-cis-v1.23-k8s-v1.25.md | 3085 ---------------- ...sessment-guide-with-cis-v1.24-k8s-v1.24.md | 3046 ++++++++++++++++ ...ide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md | 2862 +++++++++++++++ ...ssessment-guide-with-cis-v1.7-k8s-v1.25.md | 3085 ---------------- .../rke2-hardening-guide.md | 273 ++ ...sessment-guide-with-cis-v1.23-k8s-v1.23.md | 26 +- ...sessment-guide-with-cis-v1.23-k8s-v1.24.md | 3196 ---------------- ...sessment-guide-with-cis-v1.23-k8s-v1.25.md | 3196 ---------------- ...sessment-guide-with-cis-v1.24-k8s-v1.24.md | 3202 ++++++++++++++++ ...ide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md | 2967 +++++++++++++++ ...ssessment-guide-with-cis-v1.7-k8s-v1.25.md | 3196 ---------------- .../rancher-security-best-practices.md | 21 + .../rancher-security/rancher-security.md | 95 + .../rancher-webhook-hardening.md | 137 + .../security-advisories-and-cves.md | 9 + .../selinux-rpm/selinux-rpm.md | 20 + .../user-settings/user-settings.md | 19 + .../about-the-api/about-the-api.md | 93 + .../cli-with-rancher/cli-with-rancher.md | 9 + .../hardening-guides/hardening-guides.md | 55 + .../k3s-hardening-guide.md | 744 ++++ ...sessment-guide-with-cis-v1.23-k8s-v1.23.md | 34 +- ...sessment-guide-with-cis-v1.23-k8s-v1.24.md | 3148 ---------------- ...sessment-guide-with-cis-v1.23-k8s-v1.25.md | 3148 ---------------- ...sessment-guide-with-cis-v1.24-k8s-v1.24.md | 3208 ++++++++++++++++ ...ide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md | 3215 +++++++++++++++++ ...ssessment-guide-with-cis-v1.7-k8s-v1.25.md | 3148 ---------------- .../rke1-hardening-guide.md | 515 +++ ...sessment-guide-with-cis-v1.23-k8s-v1.23.md | 26 +- ...sessment-guide-with-cis-v1.23-k8s-v1.24.md | 3084 ---------------- ...sessment-guide-with-cis-v1.23-k8s-v1.25.md | 3085 ---------------- ...sessment-guide-with-cis-v1.24-k8s-v1.24.md | 3046 ++++++++++++++++ ...ide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md | 2862 +++++++++++++++ ...ssessment-guide-with-cis-v1.7-k8s-v1.25.md | 3085 ---------------- .../rke2-hardening-guide.md | 273 ++ ...sessment-guide-with-cis-v1.23-k8s-v1.23.md | 26 +- ...sessment-guide-with-cis-v1.23-k8s-v1.24.md | 3196 ---------------- ...sessment-guide-with-cis-v1.23-k8s-v1.25.md | 3196 ---------------- ...sessment-guide-with-cis-v1.24-k8s-v1.24.md | 3202 ++++++++++++++++ ...ide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md | 2967 +++++++++++++++ ...ssessment-guide-with-cis-v1.7-k8s-v1.25.md | 3196 ---------------- .../rancher-security-best-practices.md | 21 + .../rancher-security/rancher-security.md | 95 + .../rancher-webhook-hardening.md | 137 + .../security-advisories-and-cves.md | 9 + .../selinux-rpm/selinux-rpm.md | 20 + .../user-settings/user-settings.md | 19 + 61 files changed, 41082 insertions(+), 56646 deletions(-) create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/about-the-api/about-the-api.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cli-with-rancher/cli-with-rancher.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/hardening-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/rancher-security-best-practices.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/rancher-security.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/rancher-webhook-hardening.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/selinux-rpm/selinux-rpm.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/user-settings/user-settings.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/about-the-api/about-the-api.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cli-with-rancher/cli-with-rancher.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/hardening-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md delete mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/rancher-security-best-practices.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/rancher-security.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/rancher-webhook-hardening.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/selinux-rpm/selinux-rpm.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/user-settings/user-settings.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/about-the-api/about-the-api.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/about-the-api/about-the-api.md new file mode 100644 index 00000000000..f1756eb6e5d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/about-the-api/about-the-api.md @@ -0,0 +1,93 @@ +--- +title: API +--- + + + + + +## 如何使用 API + +API 有自己的用户界面，你可以从 Web 浏览器访问它。这是查看资源、执行操作以及查看等效 cURL 或 HTTP 请求和响应的一种简单的方法。要访问它： + + + + +1. 单击右上角的用户头像。 +1. 单击**账号 & API 密钥**。 +1. 在 **API 密钥**下，找到 **API 端点**字段并单击链接。该链接类似于 `https:///v3`，其中 `` 是 Rancher deployment 的完全限定域名。 + + + + +转到位于 `https:///v3` 的 URL 端点，其中 `` 是你的 Rancher deployment 的完全限定域名。 + + + + +## 认证 + +API 请求必须包含认证信息。认证是通过 [API 密钥](../user-settings/api-keys.md)使用 HTTP 基本认证完成的。API 密钥可以创建新集群并通过 `/v3/clusters/` 访问多个集群。[集群和项目角色](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md)会应用于这些键，并限制账号可以查看的集群和项目以及可以执行的操作。 + +默认情况下，某些集群级别的 API 令牌是使用无限期 TTL（`ttl=0`）生成的。换言之，除非你让令牌失效，否则 `ttl=0` 的 API 令牌永远不会过期。有关如何使 API 令牌失效的详细信息，请参阅 [API 令牌](api-tokens.md)。 + +## 发出请求 + +该 API 通常是 RESTful 的，但是还具有多种功能。这些功能可以使客户端发现所有内容，因此可以编写通用客户端，而不必为每种资源编写特定代码。有关通用 API 规范的详细信息，请参阅[此处](https://github.com/rancher/api-spec/blob/master/specification.md)。 + +- 每种类型都有一个 Schema，这个 Schema 描述了以下内容： + - 用于获取此类资源集合的 URL + - 资源可以具有的每个字段及其类型、基本验证规则、是必填还是可选字段等 + - 在此类资源上可以执行的每个操作，以及它们的输入和输出（也作为 schema） + - 允许过滤的每个字段 + - 集合本身或集合中的单个资源可以使用的 HTTP 操作方法 + + +- 因此，你可以只加载 schema 列表并了解 API 的所有信息。实际上，这是 API 的 UI 工作方式，它不包含特定于 Rancher 本身的代码。每个 HTTP 响应中的 `X-Api-Schemas` 标头都会发送获取 Schemas 的 URL。你可以按照每个 schema 上的 `collection` 链接了解要在哪里列出资源，并在返回资源中的其他 `links` 中获取其他信息。 + +- 在实践中，你可能只想构造 URL 字符串。我们强烈建议将此限制为在顶层列出的集合 (`/v3/`)，或获取特定资源 (`/v3//`)。除此之外的任何内容都可能在将来的版本中发生更改。 + +- 资源之间相互之间有联系，称为链接（links）。每个资源都包含一个 `links` 映射，其中包含链接名称和用于检索该信息的 URL。同样，你应该 `GET` 资源并遵循 `links` 映射中的 URL，而不是自己构造这些字符串。 + +- 大多数资源都有操作（action），表示可以执行某个操作或改变资源的状态。要使用操作，请将 HTTP `POST` 请求发送到 `actions` 映射中你想要的操作的 URL。某些操作需要输入或生成输出，请参阅每种类型的独立文档或 schema 以获取具体信息。 + +- 要编辑资源，请将 HTTP `PUT` 请求发送到资源上的 `links.update` 链接，其中包含要更改的字段。如果链接丢失，则你无权更新资源。未知字段和不可编辑的字段将被忽略。 + +- 要删除资源，请将 HTTP `DELETE` 请求发送到资源上的 `links.remove` 链接。如果链接丢失，则你无权更新资源。 + +- 要创建新资源，HTTP `POST` 到 schema（即 `/v3/`）中的集合 URL。 + +## 过滤 + +你可以使用 HTTP 查询参数的公共字段在服务器端过滤大多数集合。`filters` 映射显示了可以过滤的字段，以及过滤后的值在你发起的请求中是什么。API UI 具有设置过滤和显示适当请求的控件。对于简单的 "equals" 匹配，它只是 `field=value`。你可以将修饰符添加到字段名称，例如 `field_gt=42` 表示“字段大于 42”。详情请参阅 [API 规范](https://github.com/rancher/api-spec/blob/master/specification.md#filtering)。 + +## 排序 + +你可以使用 HTTP 查询参数的公共字段在服务器端排序大多数集合。`sortLinks` 映射显示了可用的排序，以及用于获取遵循该排序的集合的 URL。它还包括当前响排序依据的信息（如果指定）。 + +## 分页 + +默认情况下，API 响应以每页 100 个资源的限制进行分页。你可以通过 `limit` 查询参数进行更改，最大为 1000，例如 `/v3/pods?limit=1000`。集合响应中的 `pagination` 映射能让你知道你是否拥有完整的结果集，如果没有，则会指向下一页的链接。 + +## 捕获 Rancher API 调用 + +你可以使用浏览器开发人员工具来捕获 Rancher API 的调用方式。例如，你可以按照以下步骤使用 Chrome 开发人员工具来获取用于配置 RKE 集群的 API 调用： + +1. 在 Rancher UI 中，转到**集群管理**并单击**创建**。 +1. 单击某个集群类型。此示例使用 Digital Ocean。 +1. 使用集群名称和节点模板填写表单，但不要单击**创建**。 +1. 在创建集群之前，你需要打开开发人员工具才能看到正在记录的 API 调用。要打开工具，右键单击 Rancher UI，然后单击**检查**。 +1. 在开发者工具中，单击 **Network** 选项卡。 +1. 在 **Network** 选项卡上，确保选择了 **Fetch/XHR**。 +1. 在 Rancher UI 中，单击**创建**。在开发者工具中，你应该会看到一个名为 `cluster?_replace=true` 的新网络请求。 +1. 右键单击 `cluster?_replace=true` 并单击**复制 > 复制为 cURL**。 +1. 将结果粘贴到文本编辑器中。你将能够看到 POST 请求，包括被发送到的 URL、所有标头以及请求的完整正文。此命令可用于从命令行创建集群。请注意，请求包含凭证，因此请将请求存储在安全的地方。 + +### 启用在 API 中查看 + +你还可以查看针对各自集群和资源捕获的 Rancher API 调用。 默认情况下不启用此功能。 要启用它： + +1. 单击 UI 右上角的 **用户图标**，然后从下拉菜单中选择 **偏好设置** +1. 在**高级功能**部分下，单击**启用"在 API 中查看"** + +选中后，**在 API 中查看**链接现在将显示在 UI 资源页面上的 **⋮** 子菜单下。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cli-with-rancher/cli-with-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cli-with-rancher/cli-with-rancher.md new file mode 100644 index 00000000000..51f50ec2dca --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cli-with-rancher/cli-with-rancher.md @@ -0,0 +1,9 @@ +--- +title: Rancher CLI +--- + + + + + +Rancher CLI 是一个命令行工具，用于在工作站中与 Rancher 进行交互。以下文档将描述 [Rancher CLI](rancher-cli.md) 和 [kubectl实用程序](kubectl-utility.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cli-with-rancher/rancher-cli.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cli-with-rancher/rancher-cli.md index 4cf44238004..8e85d8dfe22 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cli-with-rancher/rancher-cli.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cli-with-rancher/rancher-cli.md @@ -3,6 +3,10 @@ title: Rancher CLI description: Rancher CLI 是一个命令行工具，用于在工作站中与 Rancher 进行交互。 --- + + + + Rancher CLI（命令行界面）是一个命令行工具，可用于与 Rancher 进行交互。使用此工具，你可以使用命令行而不用通过 GUI 来操作 Rancher。 ### 下载 Rancher CLI diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/hardening-guides.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/hardening-guides.md new file mode 100644 index 00000000000..6de7d34574f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/hardening-guides.md @@ -0,0 +1,55 @@ +--- +title: Rancher 自我评估和加固指南 +--- + + + + + +Rancher 为每个受支持的 Rancher 版本的 Kubernetes 发行版提供了特定的安全强化指南。 + +## Rancher Kubernetes 发行版 + +Rancher 使用以下 Kubernetes 发行版： + +- [**RKE**](https://rancher.com/docs/rke/latest/en/)（Rancher Kubernetes Engine）是经过 CNCF 认证的 Kubernetes 发行版，完全在 Docker 容器中运行。 +- [**RKE2**](https://docs.rke2.io/) 是一个完全合规的 Kubernetes 发行版，专注于安全和合规性。 +- [**K3s**](https://docs.k3s.io/) 是一个完全合规的，轻量级 Kubernetes 发行版。它易于安装，内存需求只有上游 Kubernetes 的一半，所有组件都在一个小于 100 MB 的二进制文件中。 + +要加固运行未列出的发行版的 Kubernetes 集群，请参阅 Kubernetes 提供商文档。 + +## 加固指南和 Benchmark 版本 + +每个自我评估指南都附有强化指南。这些指南与列出的 Rancher 版本一起进行了测试。每个自我评估指南都在特定的 Kubernetes 版本和 CIS Benchmark 版本上进行了测试。如果 CIS Benchmark 尚未针对你的 Kubernetes 版本进行验证，你可以使用现有指南，直到添加适合你的版本的指南。 + +### RKE 指南 + +| Kubernetes 版本 | CIS Benchmark 版本 | 自我评估指南 | 加固指南 | +|--------------------|-----------------------|-----------------------|------------------| +| Kubernetes v1.23 | CIS v1.23 | [链接](rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md) | [链接](rke1-hardening-guide/rke1-hardening-guide.md) | +| Kubernetes v1.24 | CIS v1.24 | [链接](rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md) | [链接](rke1-hardening-guide/rke1-hardening-guide.md) | +| Kubernetes v1.25/v1.26/v1.27 | CIS v1.7 | [链接](rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md) | [链接](rke1-hardening-guide/rke1-hardening-guide.md) | + +### RKE2 指南 + +| 类型 | Kubernetes 版本 | CIS Benchmark 版本 | 自我评估指南 | 加固指南 | +|------|--------------------|-----------------------|-----------------------|------------------| +| Rancher provisioned RKE2 | Kubernetes v1.23 | CIS v1.23 | [链接](rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md) | [链接](rke2-hardening-guide/rke2-hardening-guide.md) | +| Rancher provisioned RKE2 | Kubernetes v1.24 | CIS v1.24 | [链接](rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md) | [链接](rke2-hardening-guide/rke2-hardening-guide.md) | +| Rancher provisioned RKE2 | Kubernetes v1.25/v1.26/v1.27 | CIS v1.7 | [链接](rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md) | [链接](rke2-hardening-guide/rke2-hardening-guide.md) | +| Standalone RKE2 | Kubernetes v1.25/v1.26/v1.27 | CIS v1.7 | [链接](https://docs.rke2.io/security/cis_self_assessment123) | [链接](https://docs.rke2.io/security/hardening_guide) | + +### K3s 指南 + +| 类型 | Kubernetes 版本 | CIS Benchmark 版本 | 自我评估指南 | 加固指南 | +|------|--------------------|-----------------------|-----------------------|------------------| +| Rancher provisioned K3s cluster | Kubernetes v1.23 | CIS v1.23 | [链接](k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md) | [链接](k3s-hardening-guide/k3s-hardening-guide.md) | +| Rancher provisioned K3s cluster | Kubernetes v1.24 | CIS v1.24 | [链接](k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md) | [链接](k3s-hardening-guide/k3s-hardening-guide.md) | +| Rancher provisioned K3s cluster | Kubernetes v1.25/v1.26/v1.27 | CIS v1.7 | [链接](k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md) | [链接](k3s-hardening-guide/k3s-hardening-guide.md) | +| Standalone K3s | Kubernetes v1.22 up to v1.24 | CIS v1.23 | [链接](https://docs.k3s.io/security/self-assessment) | [链接](https://docs.k3s.io/security/hardening-guide) | + +## 在 SELinux 上使用 Rancher + +[Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) 是一个内核模块，为 Linux 添加了额外的访问控制和安全工具。SELinux 过去曾被政府机构使用，现在已成为行业标准。SELinux 在 RHEL 和 CentOS 上默认启用。 + +要将 Rancher 与 SELinux 结合使用，我们建议[安装](../selinux-rpm/about-rancher-selinux.md) `rancher-selinux`。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md new file mode 100644 index 00000000000..d40edd47e96 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md @@ -0,0 +1,744 @@ +--- +title: K3s 加固指南 +--- + + + + + +本文档提供了针对生产环境的 K3s 集群进行加固的具体指导，以便在使用 Rancher 部署之前进行配置。它概述了满足信息安全中心（Center for Information Security, CIS）Kubernetes benchmark controls 所需的配置和控制。 + +:::note +这份加固指南描述了如何确保你集群中的节点安全。我们建议你在安装 Kubernetes 之前遵循本指南。 +::: + +此加固指南适用于 K3s 集群，并与以下版本的 CIS Kubernetes Benchmark、Kubernetes 和 Rancher 相关联： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|------------------------------| +| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 至 v1.26 | + +:::note +在 Benchmark v1.7 中，不再需要 `--protect-kernel-defaults` (4.2.6) 参数，并已被 CIS 删除。 +::: + +有关如何评估加固的 K3s 集群与官方 CIS benchmark 的更多细节，请参考特定 Kubernetes 和 CIS benchmark 版本的 K3s 自我评估指南。 + +K3s 在不需要修改的情况下通过了许多 Kubernetes CIS controls，因为它默认应用了几个安全缓解措施。然而，有一些值得注意的例外情况，需要手动干预才能完全符合 CIS Benchmark 要求： + +1. K3s 不修改主机操作系统。任何主机级别的修改都需要手动完成。 +2. 某些 CIS policy controls，例如 `NetworkPolicies` 和 `PodSecurityStandards`（在 v1.24 及更早版本中为 `PodSecurityPolicies`），会限制集群功能。 + 你必须选择让 K3s 配置这些策略。在你的命令行标志或配置文件中添加相应的选项（启用准入插件），并手动应用适当的策略。 + 请参阅以下详细信息。 + +CIS Benchmark 的第一部分（1.1）主要关注于 Pod manifest 的权限和所有权。由于发行版中的所有内容都打包在一个二进制文件中，因此这一部分不适用于 K3s 的核心组件。 + +## 主机级别要求 + +### 确保 `protect-kernel-defaults`已经设置 + + + + +自 CIS benchmark 1.7 开始，不再需要`protect-kernel-defaults`。 + + + + +这是一个 kubelet 标志，如果所需的内核参数未设置或设置为与 kubelet 的默认值不同的值，将导致 kubelet 退出。 + +可以在 Rancher 的集群配置中设置 `protect-kernel-defaults` 标志。 + +```yaml +spec: + rkeConfig: + machineSelectorConfig: + - config: + protect-kernel-defaults: true +``` + + + + +### 设置内核参数 + +建议为集群中的所有节点类型设置以下 `sysctl` 配置。在 `/etc/sysctl.d/90-kubelet.conf` 中设置以下参数： + +```ini +vm.panic_on_oom=0 +vm.overcommit_memory=1 +kernel.panic=10 +kernel.panic_on_oops=1 +``` + +运行 `sudo sysctl -p /etc/sysctl.d/90-kubelet.conf` 以启用设置。 + +此配置需要在设置 kubelet 标志之前完成，否则 K3s 将无法启动。 + +## Kubernetes 运行时要求 + +CIS Benchmark 的运行时要求主要围绕 Pod 安全（通过 PSP 或 PSA）、网络策略和 API 服务器审计日志展开。 + +默认情况下，K3s 不包含任何 Pod 安全或网络策略。然而，K3s 附带一个控制器，可以强制执行你创建的任何网络策略。默认情况下，K3s 启用了 `PodSecurity` 和 `NodeRestriction` 等多个准入控制器。 + +### Pod 安全 + + + + +K3s v1.25 及更新版本支持 [Pod 安全准入(PSA)](https://kubernetes.io/docs/concepts/security/pod-security-admission/)，用于控制 Pod 安全性。 + +你可以在 Rancher 中通过集群配置，设置 `defaultPodSecurityAdmissionConfigurationTemplateName` 字段来指定 PSA 配置： + +```yaml +spec: + defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted +``` + +Rancher 提供了 `rancher-restricted` 模板，用于强制执行高度限制性的 Kubernetes 上游 [`Restricted`](https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted) 配置文件，其中包含了 Pod 加固的最佳实践。 + + + + +K3s v1.24 及更早版本支持 [Pod 安全策略 (PSP)](https://v1-24.docs.kubernetes.io/docs/concepts/security/pod-security-policy/) 以控制 Pod 安全性。 + +你可以在 Rancher 中通过集群配置，传递以下标志来启用 PSPs： + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount +``` + +这会保留 `NodeRestriction` 插件并启用 `PodSecurityPolicy`。 + +启用 PSPs 后，你可以应用策略来满足 CIS Benchmark 第 5.2 节中描述的必要控制。 + +:::note +这些是 CIS Benchmark 中的手动检查。CIS 扫描结果将标记为 `warning`，因为需要集群操作员进行手动检查。 +::: + +以下是合规的 PSP 示例： + +```yaml +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: restricted-psp +spec: + privileged: false # CIS - 5.2.1 + allowPrivilegeEscalation: false # CIS - 5.2.5 + requiredDropCapabilities: # CIS - 5.2.7/8/9 + - ALL + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'downwardAPI' + - 'csi' + - 'persistentVolumeClaim' + - 'ephemeral' + hostNetwork: false # CIS - 5.2.4 + hostIPC: false # CIS - 5.2.3 + hostPID: false # CIS - 5.2.2 + runAsUser: + rule: 'MustRunAsNonRoot' # CIS - 5.2.6 + seLinux: + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + - min: 1 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + - min: 1 + max: 65535 + readOnlyRootFilesystem: false +``` + +要使示例 PSP 生效，我们需要创建一个 `ClusterRole` 和 一个`ClusterRoleBinding`。我们还需要为需要额外权限的系统级 Pod 提供“系统无限制策略”，以及允许必要的 sysctls 来实现 ServiceLB 完整功能的额外策略。 + +```yaml +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: restricted-psp +spec: + privileged: false + allowPrivilegeEscalation: false + requiredDropCapabilities: + - ALL + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'downwardAPI' + - 'csi' + - 'persistentVolumeClaim' + - 'ephemeral' + hostNetwork: false + hostIPC: false + hostPID: false + runAsUser: + rule: 'MustRunAsNonRoot' + seLinux: + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + - min: 1 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + - min: 1 + max: 65535 + readOnlyRootFilesystem: false +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: system-unrestricted-psp + annotations: + seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*' +spec: + allowPrivilegeEscalation: true + allowedCapabilities: + - '*' + fsGroup: + rule: RunAsAny + hostIPC: true + hostNetwork: true + hostPID: true + hostPorts: + - max: 65535 + min: 0 + privileged: true + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + volumes: + - '*' +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: svclb-psp + annotations: + seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*' +spec: + allowPrivilegeEscalation: false + allowedCapabilities: + - NET_ADMIN + allowedUnsafeSysctls: + - net.ipv4.ip_forward + - net.ipv6.conf.all.forwarding + fsGroup: + rule: RunAsAny + hostPorts: + - max: 65535 + min: 0 + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:restricted-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + verbs: + - use + resourceNames: + - restricted-psp +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:system-unrestricted-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - system-unrestricted-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:svclb-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - svclb-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:svc-local-path-provisioner-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - system-unrestricted-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:svc-coredns-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - system-unrestricted-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:svc-cis-operator-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - system-unrestricted-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: default:restricted-psp +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted-psp +subjects: +- kind: Group + name: system:authenticated + apiGroup: rbac.authorization.k8s.io +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: system-unrestricted-node-psp-rolebinding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:system-unrestricted-psp +subjects: +- apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:nodes +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: system-unrestricted-svc-acct-psp-rolebinding + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:system-unrestricted-psp +subjects: +- apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: svclb-psp-rolebinding + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:svclb-psp +subjects: +- kind: ServiceAccount + name: svclb +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: svc-local-path-provisioner-psp-rolebinding + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:svc-local-path-provisioner-psp +subjects: +- kind: ServiceAccount + name: local-path-provisioner-service-account +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: svc-coredns-psp-rolebinding + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:svc-coredns-psp +subjects: +- kind: ServiceAccount + name: coredns +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: svc-cis-operator-psp-rolebinding + namespace: cis-operator-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:svc-cis-operator-psp +subjects: +- kind: ServiceAccount + name: cis-operator-serviceaccount +``` + +上述策略可以放置在 `/var/lib/rancher/k3s/server/manifests` 目录下名为 `policy.yaml` 的文件中。在启动 K3s 之前，必须创建策略文件和其目录结构。建议限制访问权限以避免泄露潜在的敏感信息。 + +```shell +sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/manifests +``` + +:::note +CNI、DNS 和 Ingress 等关键 Kubernetes 组件在 `kube-system` 命名空间中作为 Pod 运行。因此，这个命名空间的限制政策较少，从而使这些组件能够正常运行。 +::: + + + + +### 网络策略 + +CIS 要求所有命名空间应用网络策略，合理限制进入命名空间和 Pod 的流量。 + +:::note +这些是 CIS Benchmark 中的手动检查。CIS 扫描结果将标记为 `warning`，因为需要集群操作员进行手动检查。 +::: + +网络策略可以放置在 `/var/lib/rancher/k3s/server/manifests` 目录下的 `policy.yaml` 文件中。如果该目录不是作为 PSP（如上所述）的一部分创建的，则必须首先创建该目录。 + +```shell +sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/manifests +``` + +以下是合规的网络策略示例： + +```yaml +--- +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: intra-namespace + namespace: kube-system +spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + name: kube-system +--- +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: intra-namespace + namespace: default +spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + name: default +--- +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: intra-namespace + namespace: kube-public +spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + name: kube-public +``` + +除非特意允许，否则活动限制会阻止 DNS。以下是允许 DNS 相关流量的网络策略示例： + +```yaml +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-network-dns-policy + namespace: +spec: + ingress: + - ports: + - port: 53 + protocol: TCP + - port: 53 + protocol: UDP + podSelector: + matchLabels: + k8s-app: kube-dns + policyTypes: + - Ingress +``` + +如果没有创建网络策略来允许访问，则默认情况下会阻止 metrics-server 和 Traefik Ingress 控制器。 + +```yaml +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-all-metrics-server + namespace: kube-system +spec: + podSelector: + matchLabels: + k8s-app: metrics-server + ingress: + - {} + policyTypes: + - Ingress +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-all-svclbtraefik-ingress + namespace: kube-system +spec: + podSelector: + matchLabels: + svccontroller.k3s.cattle.io/svcname: traefik + ingress: + - {} + policyTypes: + - Ingress +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-all-traefik-v121-ingress + namespace: kube-system +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: traefik + ingress: + - {} + policyTypes: + - Ingress +``` + +:::note +你必须像平常一样管理你创建的任何其他命名空间的网络策略。 +::: + +### API server 审计配置 + +CIS 要求 1.2.19 至 1.2.22 与配置 API server 审核日志相关。默认情况下，K3s 不会创建日志目录和审计策略，因为每个用户的审计策略要求和环境都是特定的。 + +如果你需要日志目录，则必须在启动 K3s 之前创建它。我们建议限制访问权限以避免泄露敏感信息。 + +```bash +sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/logs +``` + +以下是用于记录请求元数据的初始审计策略。应将策略写入到 `/var/lib/rancher/k3s/server` 目录下名为 `audit.yaml` 的文件中。有关 API server 的策略配置的详细信息，请参阅 [官方 Kubernetes 文档](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/)。 + +```yaml +--- +apiVersion: audit.k8s.io/v1 +kind: Policy +rules: +- level: Metadata +``` + +还需要进一步配置才能通过 CIS 检查。这些在 K3s 中默认不配置，因为它们根据你的环境和需求而有所不同： + +- 确保 `--audit-log-path` 参数已经设置。 +- 确保 `--audit-log-maxage` 参数设置为 30 或适当的值。 +- 确保 `--audit-log-maxbackup` 参数设置为 10 或适当的值。 +- 确保 `--audit-log-maxsize` 参数设置为 100 或适当的值。 + +综合起来，要启用和配置审计日志，请将以下行添加到 Rancher 的 K3s 集群配置文件中： + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS 3.2.1 + - audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS 1.2.18 + - audit-log-maxage=30 # CIS 1.2.19 + - audit-log-maxbackup=10 # CIS 1.2.20 + - audit-log-maxsize=100 # CIS 1.2.21 +``` + +### Controller Manager 要求 + +CIS 要求 1.3.1 检查 Controller Manager 中的垃圾收集设置。垃圾收集对于确保资源充足可用性并避免性能和可用性下降非常重要。根据你的系统资源和测试结果，选择一个适当的阈值来激活垃圾收集。 + +你可以在 Rancher 的 K3s 集群文件中设置以下配置来解决此问题。下面的值仅是一个示例，请根据当前环境设置适当的阈值。 + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + kube-controller-manager-arg: + - terminated-pod-gc-threshold=10 # CIS 1.3.1 +``` + +### 配置 `default` Service Account + +Kubernetes 提供了一个名为 `default` 的 service account，供集群工作负载使用，其中没有为 Pod 分配特定的 service account。当 Pod 需要从 Kubernetes API 获取访问权限时，应为该 Pod 创建一个特定的 service account，并为该 service account 授予权限。 + +对于 CIS 5.1.5，`default` service account 应配置为不提供 service account 令牌，并且不具有任何明确的权限分配。 + +可以通过在每个命名空间中将 `default` service account 的 `automountServiceAccountToken` 字段更新为 `false` 来解决此问题。 + +对于内置命名空间（`kube-system`、`kube-public`、`kube-node-lease` 和 `default`）中的 `default` service accounts，K3s 不会自动执行此操作。 + +将以下配置保存到名为 `account_update.yaml` 的文件中。 + +```yaml +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: default +automountServiceAccountToken: false +``` + +创建一个名为 `account_update.sh` 的 Bash 脚本文件。确保使用 `chmod +x account_update.sh` 给脚本添加可执行权限。 + +```shell +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" +done +``` + +每次向你的集群添加新的 service account 时，运行该脚本。 + +## 加固版 K3s 模板配置参考 + +Rancher 使用以下参考模板配置，基于本指南中的每个 CIS 控件创建加固过的自定义 K3s 集群。此参考内容不包括其他必需的**集群配置**指令，这些指令因你的环境而异。 + + + + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: # 定义集群名称 +spec: + defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted + enableNetworkPolicy: true + kubernetesVersion: # 定义 K3s 版本 + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - enable-admission-plugins=NodeRestriction,ServiceAccount # CIS 1.2.15, 1.2.13 + - audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS 3.2.1 + - audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS 1.2.18 + - audit-log-maxage=30 # CIS 1.2.19 + - audit-log-maxbackup=10 # CIS 1.2.20 + - audit-log-maxsize=100 # CIS 1.2.21 + - request-timeout=300s # CIS 1.2.22 + - service-account-lookup=true # CIS 1.2.24 + kube-controller-manager-arg: + - terminated-pod-gc-threshold=10 # CIS 1.3.1 + secrets-encryption: true + machineSelectorConfig: + - config: + kubelet-arg: + - make-iptables-util-chains=true # CIS 4.2.7 +``` + + + + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: # 定义集群名称 +spec: + enableNetworkPolicy: true + kubernetesVersion: # 定义 K3s 版本 + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount # CIS 1.2.15, 5.2, 1.2.13 + - audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS 3.2.1 + - audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS 1.2.18 + - audit-log-maxage=30 # CIS 1.2.19 + - audit-log-maxbackup=10 # CIS 1.2.20 + - audit-log-maxsize=100 # CIS 1.2.21 + - request-timeout=300s # CIS 1.2.22 + - service-account-lookup=true # CIS 1.2.24 + kube-controller-manager-arg: + - terminated-pod-gc-threshold=10 # CIS 1.3.1 + secrets-encryption: true + machineSelectorConfig: + - config: + kubelet-arg: + - make-iptables-util-chains=true # CIS 4.2.7 + protect-kernel-defaults: true # CIS 4.2.6 +``` + + + + +## 结论 + +如果你按照本指南操作，由 Rancher 提供的 K3s 自定义集群将配置为通过 CIS Kubernetes Benchmark 测试。你可以查看我们的 K3s 自我评估指南，了解我们是如何验证每个 benchmarks 的，并且你可以在你的集群上执行相同的操作。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md index ddf9809440a..5bfb97c99f7 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md @@ -1,36 +1,40 @@ --- -title: K3s Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.23 +title: K3s 自我评估指南 - CIS Benchmark v1.23 - K8s v1.23 --- -This document is a companion to the [K3s Hardening Guide](../../../../pages-for-subheaders/k3s-hardening-guide.md), which provides prescriptive guidance on how to harden K3s clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. + + + -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: +本文档是 [K3s 加固指南](k3s-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 K3s 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 -| Rancher Version | CIS Benchmark Version | Kubernetes Version | +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | |-----------------|-----------------------|--------------------| | Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | -This document is for Rancher operators, security teams, auditors and decision makers. +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.23 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 -## Testing Methodology +## 测试方法 -Each control in the CIS Kubernetes Benchmark was evaluated against a K3s cluster that was configured according to the accompanying hardening guide. +每个 CIS Kubernetes Benchmark 中的 control 都根据附带的加固指南评估了针对 K3s 集群的配置。 -Where control audits differ from the original CIS benchmark, the audit commands specific to K3s are provided for testing. +当 control 审计与原始的 CIS benchmark 不同的时候，提供了针对 K3s 的特定审计命令，以供测试使用。 -These are the possible results for each control: +以下是每个 control 可能的结果： -- **Pass** - The K3s cluster passes the audit outlined in the benchmark. -- **Not Applicable** - The control is not applicable to K3s because of how it is designed to operate. The remediation section explains why. -- **Warn** - The control is manual in the CIS benchmark and it depends on the cluster's use-case or some other factor that must be determined by the cluster operator. These controls have been evaluated to ensure K3s doesn't prevent their implementation, but no further configuration or auditing of the cluster has been performed. +- **Pass（通过）** - K3s 集群通过了 benchmark 中概述的审计。 +- **Not Applicable（不适用）** - 由于 K3s 的设计方式，该 control 不适用于 K3s。在补救措施部分解释了原因。 +- **Warn（警告）** - 在 CIS benchmark 中，该 control 是手动的，它取决于集群的使用情况或其他必须由集群操作员确定的因素。这些 control 措施已经过评估，以确保 K3s 不会阻止其实施，但尚未对集群进行进一步的配置或审计。 -This guide makes the assumption that K3s is running as a Systemd unit. Your installation may vary. Adjust the "audit" commands to fit your scenario. +本指南假设 K3s 作为 Systemd 单元运行。你的安装可能会有所不同。调整"审计"命令以适合你的场景。 :::note -This guide only covers `automated` (previously called `scored`) tests. +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md deleted file mode 100644 index 8944de04c27..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md +++ /dev/null @@ -1,3148 +0,0 @@ ---- -title: K3s Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.24 ---- - -This document is a companion to the [K3s Hardening Guide](../../../../pages-for-subheaders/k3s-hardening-guide.md), which provides prescriptive guidance on how to harden K3s clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.24 | - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -Each control in the CIS Kubernetes Benchmark was evaluated against a K3s cluster that was configured according to the accompanying hardening guide. - -Where control audits differ from the original CIS benchmark, the audit commands specific to K3s are provided for testing. - -These are the possible results for each control: - -- **Pass** - The K3s cluster passes the audit outlined in the benchmark. -- **Not Applicable** - The control is not applicable to K3s because of how it is designed to operate. The remediation section explains why. -- **Warn** - The control is manual in the CIS benchmark and it depends on the cluster's use-case or some other factor that must be determined by the cluster operator. These controls have been evaluated to ensure K3s doesn't prevent their implementation, but no further configuration or auditing of the cluster has been performed. - -This guide makes the assumption that K3s is running as a Systemd unit. Your installation may vary. Adjust the "audit" commands to fit your scenario. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Control Plane Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the -control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-apiserver.yaml - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-controller-manager.yaml - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-scheduler.yaml - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 /etc/kubernetes/manifests/etcd.yaml - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /etc/kubernetes/manifests/etcd.yaml - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 1.1.11 -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** Not Applicable - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/admin.conf - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 scheduler - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root scheduler - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 controllermanager - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/controller.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root controllermanager - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown -R root:root /etc/kubernetes/pki/ - -**Audit:** - -```bash -find /var/lib/rancher/k3s/server/tls | xargs stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) - - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - -**Audit:** - -```bash -stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) - - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /etc/kubernetes/pki/*.key - -**Audit:** - -```bash -stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep containerd | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' is present OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 521 1 0 22:09 ? 00:00:00 /usr/bin/containerd root 802 1 0 22:09 ? 00:00:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 3970 1 0 22:13 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id 836d5508918bf22689e12f2885aeb9de5c2420beb956f6dc587d11b049e4edf4 -address /run/k3s/containerd/containerd.sock root 4088 1 0 22:13 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id 6148f976b57db791da86119c599700f683df838df56b1382d428ec38402caa64 -address /run/k3s/containerd/containerd.sock root 4109 1 0 22:13 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id b0e22b39e80606c4ad5d7fdafe21a771a27a5dce8f06155af579a9fad8158219 -address /run/k3s/containerd/containerd.sock root 5321 1 0 22:14 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id f60b96ef6beaf1b7903aba944af101a7d401082969ad8b5aea633bda1043b165 -address /run/k3s/containerd/containerd.sock root 5395 1 0 22:14 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id df001d7d85239f1cfdd8e041db801a6383bc18e5e10d0cd8e8e4e6ca9b7ce036 -address /run/k3s/containerd/containerd.sock root 6366 1 0 22:15 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id a900df9f736ca35b5dbd603034588a461d706ae6c83529136a81ec13e0ecedc0 -address /run/k3s/containerd/containerd.sock root 7859 1 0 22:15 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id b3c1510cafad33b31758db1b7c771efe98f0d395d5d52caf90db7bd379966299 -address /run/k3s/containerd/containerd.sock root 7896 1 0 22:15 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id e5720fc8f9e95f2fd8b8e87c93f25213315d78b630fc889b2a5cb8b18f2a1442 -address /run/k3s/containerd/containerd.sock root 9773 9761 4 22:22 ? 00:00:16 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 10275 1 0 22:22 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id 8b29d2205c5f7d47d0ecd4f95b7b96062454233e66450f9163058647e7b5c390 -address /run/k3s/containerd/containerd.sock root 11185 1 0 22:23 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9581871a145dd1c0bbff06cddea0030a30de2760e90c5fa9154cccb250f05faa -address /run/k3s/containerd/containerd.sock root 13605 1 0 22:27 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id e05825dc2a33d576c35cc1b56610d3e4459d209524645f95f163c100f6e5df46 -address /run/k3s/containerd/containerd.sock root 13770 1 0 22:27 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id 74d10151f05f78b0d38deb2b1ee045bf7b72e7b879e6a15112c8495f98d230b8 -address /run/k3s/containerd/containerd.sock -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'AlwaysPullImages' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'ServiceAccount' -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-path' -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxage' -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxbackup' -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxsize' -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.23 Ensure that the --request-timeout argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameter as appropriate and if needed. -For example, --request-timeout=300s - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'request-timeout' -``` - -**Expected Result**: - -```console -'--request-timeout' is not present OR '--request-timeout' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-lookup' -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.25 Ensure that the --service-account-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-key-file' -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 1.2.29 -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'encryption-provider-config' -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.31 Ensure that encryption providers are appropriately configured (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit:** - -```bash -grep aescbc /path/to/encryption-config.json -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) - - -**Result:** warn - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'bind-address' -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.1 -``` - -**Expected Result**: - -```console -'cert-file' is present AND 'key-file' is present -``` - -**Returned Value**: - -```console -cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key -``` - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and set the below parameter. ---client-cert-auth="true" - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.2 -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -client-cert-auth: true -``` - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.3 -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.4 -``` - -**Expected Result**: - -```console -'cert-file' is present AND 'key-file' is present -``` - -**Returned Value**: - -```console -cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key -``` - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and set the below parameter. ---peer-client-cert-auth=true - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.5 -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -client-cert-auth: true -``` - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.6 -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.7 -``` - -**Expected Result**: - -```console -'trusted-ca-file' is present -``` - -**Returned Value**: - -```console -trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Manual) - - -**Result:** warn - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig -``` - -**Expected Result**: - -```console -'permissions' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/agent/kubelet.kubeconfig -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt -``` - -**Expected Result**: - -```console -'644' is equal to '644' OR '640' is present OR '600' is present OR '444' is present OR '440' is present OR '400' is present OR '000' is present -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" | grep -v grep; else echo "--anonymous-auth=false"; fi' -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" | grep -v grep; else echo "--authorization-mode=Webhook"; fi' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" | grep -v grep; else echo "--client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt"; fi' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:03 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:03Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-2 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=995abdc2-f967-4bd6-936e-f3c26594573c --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'protect-kernel-defaults' -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -Feb 27 22:22:03 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:03Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-2 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=995abdc2-f967-4bd6-936e-f3c26594573c --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' -``` - -**Expected Result**: - -```console -'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:03 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:03Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-2 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=995abdc2-f967-4bd6-936e-f3c26594573c --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:03 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:03Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-2 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=995abdc2-f967-4bd6-936e-f3c26594573c --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Manual) - - -**Result:** warn - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** warn - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) - - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md deleted file mode 100644 index 1f5b7dfd37b..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md +++ /dev/null @@ -1,3148 +0,0 @@ ---- -title: K3s Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.25 ---- - -This document is a companion to the [K3s Hardening Guide](../../../../pages-for-subheaders/k3s-hardening-guide.md), which provides prescriptive guidance on how to harden K3s clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.25 | - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -Each control in the CIS Kubernetes Benchmark was evaluated against a K3s cluster that was configured according to the accompanying hardening guide. - -Where control audits differ from the original CIS benchmark, the audit commands specific to K3s are provided for testing. - -These are the possible results for each control: - -- **Pass** - The K3s cluster passes the audit outlined in the benchmark. -- **Not Applicable** - The control is not applicable to K3s because of how it is designed to operate. The remediation section explains why. -- **Warn** - The control is manual in the CIS benchmark and it depends on the cluster's use-case or some other factor that must be determined by the cluster operator. These controls have been evaluated to ensure K3s doesn't prevent their implementation, but no further configuration or auditing of the cluster has been performed. - -This guide makes the assumption that K3s is running as a Systemd unit. Your installation may vary. Adjust the "audit" commands to fit your scenario. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Control Plane Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the -control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-apiserver.yaml - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-controller-manager.yaml - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-scheduler.yaml - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 /etc/kubernetes/manifests/etcd.yaml - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /etc/kubernetes/manifests/etcd.yaml - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 1.1.11 -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** Not Applicable - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/admin.conf - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 scheduler - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root scheduler - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 controllermanager - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/controller.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root controllermanager - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown -R root:root /etc/kubernetes/pki/ - -**Audit:** - -```bash -find /var/lib/rancher/k3s/server/tls | xargs stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) - - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - -**Audit:** - -```bash -stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) - - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /etc/kubernetes/pki/*.key - -**Audit:** - -```bash -stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep containerd | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' is present OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 519 1 0 22:09 ? 00:00:00 /usr/bin/containerd root 801 1 0 22:09 ? 00:00:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 3864 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id d00174abbc275f6bb85c7f0be1d3154b9c91982a10b9dba6b5cb280f4d4c531d -address /run/k3s/containerd/containerd.sock root 4105 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7c2b546b4d2380bcb51278661f34cff94fad2ba06978e13f8f1b92dafcc89d43 -address /run/k3s/containerd/containerd.sock root 4206 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 68d8a55ff4663985be004608dbf78b0362f5522e18490c81d4c8dc9963de1556 -address /run/k3s/containerd/containerd.sock root 5374 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id ca0ae9e0b37dfd7b1ce05f72e1bc5a1be8f5cb08f2b4543081536de3bdbc925d -address /run/k3s/containerd/containerd.sock root 5443 1 0 22:31 ? 00:00:01 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3ea3c1cdbbd5adb8efd5c67a46aadd0fca9918dc0ad1f7cafe38b83171e3dc1b -address /run/k3s/containerd/containerd.sock root 7130 1 0 22:32 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4d838297d35a31003106ac5989c3547433985bb2964b47baad12cee6e375645e -address /run/k3s/containerd/containerd.sock root 7639 1 0 22:32 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 341cb9bcd8486aa2f1acb8e1ae51baebd630ac6ed266643266c34d677f61c7d0 -address /run/k3s/containerd/containerd.sock root 10308 1 0 23:17 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id c534fbee8e0d06fd9b29bf8fc70a138975c6b18db25f1faf2615677dfdb4199e -address /run/k3s/containerd/containerd.sock root 11370 1 0 23:18 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4ff4b8776dac7a35b83616d341dbe4d5a689ac7fb9b8eee8db5978e3968380ea -address /run/k3s/containerd/containerd.sock root 13736 13723 2 23:21 ? 00:00:10 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 16022 1 0 23:29 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9027256349086e458119478e5e00384b1b76fbf5e6dbee23699f596a88d9f2bc -address /run/k3s/containerd/containerd.sock root 16159 1 0 23:29 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 929bf369fc5881654f4c1925624151ddb7cea51073267b8d213d966ba45406f3 -address /run/k3s/containerd/containerd.sock -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'AlwaysPullImages' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'ServiceAccount' -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-path' -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxage' -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxbackup' -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxsize' -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.23 Ensure that the --request-timeout argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameter as appropriate and if needed. -For example, --request-timeout=300s - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'request-timeout' -``` - -**Expected Result**: - -```console -'--request-timeout' is not present OR '--request-timeout' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-lookup' -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.25 Ensure that the --service-account-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-key-file' -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 1.2.29 -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'encryption-provider-config' -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.31 Ensure that encryption providers are appropriately configured (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit:** - -```bash -grep aescbc /path/to/encryption-config.json -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) - - -**Result:** warn - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'bind-address' -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.1 -``` - -**Expected Result**: - -```console -'cert-file' is present AND 'key-file' is present -``` - -**Returned Value**: - -```console -cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key -``` - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and set the below parameter. ---client-cert-auth="true" - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.2 -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -client-cert-auth: true -``` - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.3 -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.4 -``` - -**Expected Result**: - -```console -'cert-file' is present AND 'key-file' is present -``` - -**Returned Value**: - -```console -cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key -``` - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and set the below parameter. ---peer-client-cert-auth=true - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.5 -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -client-cert-auth: true -``` - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.6 -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.7 -``` - -**Expected Result**: - -```console -'trusted-ca-file' is present -``` - -**Returned Value**: - -```console -trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Manual) - - -**Result:** warn - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig -``` - -**Expected Result**: - -```console -'permissions' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/agent/kubelet.kubeconfig -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt -``` - -**Expected Result**: - -```console -'644' is equal to '644' OR '640' is present OR '600' is present OR '444' is present OR '440' is present OR '400' is present OR '000' is present -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" | grep -v grep; else echo "--anonymous-auth=false"; fi' -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" | grep -v grep; else echo "--authorization-mode=Webhook"; fi' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" | grep -v grep; else echo "--client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt"; fi' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'protect-kernel-defaults' -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' -``` - -**Expected Result**: - -```console -'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Manual) - - -**Result:** warn - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** warn - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) - - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md new file mode 100644 index 00000000000..464073a825f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md @@ -0,0 +1,3208 @@ +--- +title: K3s 自我评估指南 - CIS Benchmark v1.24 - K8s v1.24 +--- + + + + + +本文档是 [K3s 加固指南](k3s-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 K3s 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.24 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +每个 CIS Kubernetes Benchmark 中的 control 都根据附带的加固指南评估了针对 K3s 集群的配置。 + +当 control 审计与原始的 CIS benchmark 不同的时候，提供了针对 K3s 的特定审计命令，以供测试使用。 + +以下是每个 control 可能的结果： + +- **Pass（通过）** - K3s 集群通过了 benchmark 中概述的审计。 +- **Not Applicable（不适用）** - 由于 K3s 的设计方式，该 control 不适用于 K3s。在补救措施部分解释了原因。 +- **Warn（警告）** - 在 CIS benchmark 中，该 control 是手动的，它取决于集群的使用情况或其他必须由集群操作员确定的因素。这些 control 措施已经过评估，以确保 K3s 不会阻止其实施，但尚未对集群进行进一步的配置或审计。 + +本指南假设 K3s 作为 Systemd 单元运行。你的安装可能会有所不同。调整"审计"命令以适合你的场景。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 644 /etc/kubernetes/manifests/kube-apiserver.yaml + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-controller-manager.yaml + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-scheduler.yaml + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /etc/kubernetes/manifests/etcd.yaml + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /etc/kubernetes/manifests/etcd.yaml + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.1.11 +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** Not Applicable + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 scheduler + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root scheduler + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 controllermanager + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/controller.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root controllermanager + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/tls +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit:** + +```bash +find /var/lib/rancher/k3s/server/tls | xargs stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.crt + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.key + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' is present OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 410 1 0 Sep11 ? 00:01:50 /usr/bin/containerd root 539 1 0 Sep11 ? 00:00:09 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 45213 45195 3 Sep11 ? 00:45:14 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 47188 1 0 Sep11 ? 00:01:00 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 02b8fdd94b7628d575ace92f337dcc93202151d07f33858341cfcb0178fea586 -address /run/k3s/containerd/containerd.sock root 47235 1 0 Sep11 ? 00:00:33 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id eb0f03c4bc1125ae34e5de77c7eaeb9c2c1e46bec0bb106b99281df7353f9ded -address /run/k3s/containerd/containerd.sock root 47948 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5eece7110bd3876414f890ed1b7158eb91f60d1f14450a9f8c77a0fc62b73c4a -address /run/k3s/containerd/containerd.sock root 48047 1 0 Sep11 ? 00:00:32 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4ef831f4bb8830a0a7b46a1e8be58197be00846ea0caa094014ea7ce3adf008e -address /run/k3s/containerd/containerd.sock root 48220 1 0 Sep11 ? 00:00:33 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id f8d5d3791ad2b1a8534476aa2fbdf8da1e4f0b9fdcdd26b1ae161142458a5336 -address /run/k3s/containerd/containerd.sock root 48878 1 0 Sep11 ? 00:00:32 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 41b93a6581e272b0852257b3d7c44ab4cadcdf465a3579a8b29b493ea75feaf7 -address /run/k3s/containerd/containerd.sock root 49870 1 0 Sep11 ? 00:00:30 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id be329cdccda4049f048c759a2b1915cd72526473ff7768e895bcfc153a66c125 -address /run/k3s/containerd/containerd.sock root 50271 1 0 Sep11 ? 00:00:32 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id cceb5d26eba7d7d3c474a2dc842b72495689f4e40055505ea2598e4ac6849d40 -address /run/k3s/containerd/containerd.sock root 50571 1 0 Sep11 ? 00:00:32 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3a26e691e0d0186165d523bef22b0af71f175c39df82845b32ce7b4e3d652d4a -address /run/k3s/containerd/containerd.sock root 96693 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9ed77e6e58410f1295f730ba4a8f8daf7aff8f4e896e24219a8fa710729c2287 -address /run/k3s/containerd/containerd.sock root 97726 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 1be4fe50e6bb29d4549f8b03102b5dc15b7669fe56b32beae3f701cf6d2e58cc -address /run/k3s/containerd/containerd.sock root 97923 1 1 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7c65c47376f36a8980a5bebad7ffdc1abe1853821cf6c4a1d45f1b6be3452062 -address /run/k3s/containerd/containerd.sock +``` + +### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --kubelet-https parameter. + +### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'ServiceAccount' +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.18 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-path' +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxage' +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxbackup' +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxsize' +``` + +**Expected Result**: + +```console +'--audit-log-maxsize' is greater or equal to 100 +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.23 Ensure that the --request-timeout argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, --request-timeout=300s + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'request-timeout' +``` + +**Expected Result**: + +```console +'--request-timeout' is not present OR '--request-timeout' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-lookup' +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.25 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-key-file' +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.2.29 +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'encryption-provider-config' +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.31 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit:** + +```bash +grep aescbc /path/to/encryption-config.json +``` + +### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'bind-address' +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.1 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.2 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.3 +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.4 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.5 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.6 +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.7 +``` + +**Expected Result**: + +```console +'trusted-ca-file' is present +``` + +**Returned Value**: + +```console +trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Manual) + + +**Result:** warn + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +**Returned Value**: + +```console +600 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'600' is equal to '600' +``` + +**Returned Value**: + +```console +600 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +**Returned Value**: + +```console +644 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/kubelet/config.yaml + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" | grep -v grep; else echo "--anonymous-auth=false"; fi' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" | grep -v grep; else echo "--authorization-mode=Webhook"; fi' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" | grep -v grep; else echo "--client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt"; fi' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:32:02 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:32:02Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-3-32 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=cc5faddd-9c87-4a04-97c2-dcf4cbcbe10f --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' +``` + +### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'protect-kernel-defaults' +``` + +**Expected Result**: + +```console +'--protect-kernel-defaults' is equal to 'true' +``` + +**Returned Value**: + +```console +Sep 11 20:32:02 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:32:02Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-3-32 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=cc5faddd-9c87-4a04-97c2-dcf4cbcbe10f --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' +``` + +**Expected Result**: + +```console +'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:32:02 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:32:02Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-3-32 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=cc5faddd-9c87-4a04-97c2-dcf4cbcbe10f --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +### 4.2.9 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'--event-qps' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 410 1 0 Sep11 ? 00:01:50 /usr/bin/containerd root 45213 45195 3 Sep11 ? 00:45:15 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:32:02 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:32:02Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-3-32 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=cc5faddd-9c87-4a04-97c2-dcf4cbcbe10f --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 410 1 0 Sep11 ? 00:01:50 /usr/bin/containerd root 45213 45195 3 Sep11 ? 00:45:15 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 410 1 0 Sep11 ? 00:01:50 /usr/bin/containerd root 45213 45195 3 Sep11 ? 00:45:15 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** warn + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +### 5.2.7 Minimize the admission of root containers (Automated) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Automated) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md new file mode 100644 index 00000000000..8443a80b530 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md @@ -0,0 +1,3215 @@ +--- +title: K3s 自我评估指南 - CIS Benchmark v1.7 - K8s v1.25/v1.26/v1.27 +--- + + + + + +本文档是 [K3s 加固指南](k3s-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 K3s 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25/v1.26/v1.27 | + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.7 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +每个 CIS Kubernetes Benchmark 中的 control 都根据附带的加固指南评估了针对 K3s 集群的配置。 + +当 control 审计与原始的 CIS benchmark 不同的时候，提供了针对 K3s 的特定审计命令，以供测试使用。 + +以下是每个 control 可能的结果： + +- **Pass（通过）** - K3s 集群通过了 benchmark 中概述的审计。 +- **Not Applicable（不适用）** - 由于 K3s 的设计方式，该 control 不适用于 K3s。在补救措施部分解释了原因。 +- **Warn（警告）** - 在 CIS benchmark 中，该 control 是手动的，它取决于集群的使用情况或其他必须由集群操作员确定的因素。这些 control 措施已经过评估，以确保 K3s 不会阻止其实施，但尚未对集群进行进一步的配置或审计。 + +本指南假设 K3s 作为 Systemd 单元运行。你的安装可能会有所不同。调整"审计"命令以适合你的场景。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-apiserver.yaml +Not Applicable. + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml +Not Applicable. + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-controller-manager.yaml +Not Applicable. + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml +Not Applicable. + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-scheduler.yaml +Not Applicable. + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml +Not Applicable. + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /etc/kubernetes/manifests/etcd.yaml +Not Applicable. + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /etc/kubernetes/manifests/etcd.yaml +Not Applicable. + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 +Not Applicable. + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root +Not Applicable. + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.1.11 +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** Not Applicable + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd +Not Applicable. + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 scheduler + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root scheduler + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 controllermanager + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/controller.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root controllermanager + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/cred/controller.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /var/lib/rancher/k3s/server/tls + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/tls +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.crt + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.key + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 663 1 0 Sep11 ? 00:00:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 3021 1 0 Sep11 ? 00:00:29 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4790d392966915d995e666002c56ed4cce6dd86f305ce2ee390547a1fcbf6c82 -address /run/k3s/containerd/containerd.sock root 3035 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9d73e6a160ccde6c7c7d4ba0df3b9f696e3de2ebfc0f19a1dcbdf13aea496427 -address /run/k3s/containerd/containerd.sock root 3235 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 587f6221ee9f36c877231ada8d816a799dfda186332c33378d1eb16c72cdc87d -address /run/k3s/containerd/containerd.sock root 4435 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a74c9ec7d99785c2f2d4e6826aa80c22eb8b38249e8f99679ece00a818e9b7b3 -address /run/k3s/containerd/containerd.sock root 4985 1 0 Sep11 ? 00:00:53 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b0c9784dbe0fcbe8be10c857976e70ec84a208cb814e87b5ca085a02d434f8c -address /run/k3s/containerd/containerd.sock root 5056 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id f3231ff35f18056e74eda14907f296be15f7ea1c6ae5ab7904e27d4d18183301 -address /run/k3s/containerd/containerd.sock root 5868 1 0 Sep11 ? 00:00:27 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3e908c4d0b10df275bdef6f72fbcfa09517d11cf749236ad020364e14d77bc93 -address /run/k3s/containerd/containerd.sock root 6158 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id bec5780a5c73fa3154aa3b5ee26cdf23202db821205893e7e66ae17e6103e97b -address /run/k3s/containerd/containerd.sock root 7366 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a81b78845bdcaef710314c93e5ea0d0617f37a8929472f7b570ab90c6667f57f -address /run/k3s/containerd/containerd.sock root 97274 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id c90652c935d9e79af45b9a9ac2b4fe315e2a761a0525737fdb95c680123a164c -address /run/k3s/containerd/containerd.sock root 98309 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 431a2763636488efd7d104c228d41f4f2ecd8c06a7fc375d8977ab0d238936a8 -address /run/k3s/containerd/containerd.sock root 98493 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b480d2fa55c8cc0105ec1902887040c017f03d5b1eb6c73c24bb9d523ad9b37 -address /run/k3s/containerd/containerd.sock +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' is present OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 663 1 0 Sep11 ? 00:00:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 3021 1 0 Sep11 ? 00:00:29 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4790d392966915d995e666002c56ed4cce6dd86f305ce2ee390547a1fcbf6c82 -address /run/k3s/containerd/containerd.sock root 3035 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9d73e6a160ccde6c7c7d4ba0df3b9f696e3de2ebfc0f19a1dcbdf13aea496427 -address /run/k3s/containerd/containerd.sock root 3235 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 587f6221ee9f36c877231ada8d816a799dfda186332c33378d1eb16c72cdc87d -address /run/k3s/containerd/containerd.sock root 4435 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a74c9ec7d99785c2f2d4e6826aa80c22eb8b38249e8f99679ece00a818e9b7b3 -address /run/k3s/containerd/containerd.sock root 4985 1 0 Sep11 ? 00:00:53 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b0c9784dbe0fcbe8be10c857976e70ec84a208cb814e87b5ca085a02d434f8c -address /run/k3s/containerd/containerd.sock root 5056 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id f3231ff35f18056e74eda14907f296be15f7ea1c6ae5ab7904e27d4d18183301 -address /run/k3s/containerd/containerd.sock root 5868 1 0 Sep11 ? 00:00:27 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3e908c4d0b10df275bdef6f72fbcfa09517d11cf749236ad020364e14d77bc93 -address /run/k3s/containerd/containerd.sock root 6158 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id bec5780a5c73fa3154aa3b5ee26cdf23202db821205893e7e66ae17e6103e97b -address /run/k3s/containerd/containerd.sock root 7366 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a81b78845bdcaef710314c93e5ea0d0617f37a8929472f7b570ab90c6667f57f -address /run/k3s/containerd/containerd.sock root 97274 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id c90652c935d9e79af45b9a9ac2b4fe315e2a761a0525737fdb95c680123a164c -address /run/k3s/containerd/containerd.sock root 98309 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 431a2763636488efd7d104c228d41f4f2ecd8c06a7fc375d8977ab0d238936a8 -address /run/k3s/containerd/containerd.sock root 98493 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b480d2fa55c8cc0105ec1902887040c017f03d5b1eb6c73c24bb9d523ad9b37 -address /run/k3s/containerd/containerd.sock +``` + +### 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= +Permissive - When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. + +### 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.7 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.9 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' is present +``` + +**Returned Value**: + +```console +root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 663 1 0 Sep11 ? 00:00:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 3021 1 0 Sep11 ? 00:00:29 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4790d392966915d995e666002c56ed4cce6dd86f305ce2ee390547a1fcbf6c82 -address /run/k3s/containerd/containerd.sock root 3035 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9d73e6a160ccde6c7c7d4ba0df3b9f696e3de2ebfc0f19a1dcbdf13aea496427 -address /run/k3s/containerd/containerd.sock root 3235 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 587f6221ee9f36c877231ada8d816a799dfda186332c33378d1eb16c72cdc87d -address /run/k3s/containerd/containerd.sock root 4435 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a74c9ec7d99785c2f2d4e6826aa80c22eb8b38249e8f99679ece00a818e9b7b3 -address /run/k3s/containerd/containerd.sock root 4985 1 0 Sep11 ? 00:00:53 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b0c9784dbe0fcbe8be10c857976e70ec84a208cb814e87b5ca085a02d434f8c -address /run/k3s/containerd/containerd.sock root 5056 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id f3231ff35f18056e74eda14907f296be15f7ea1c6ae5ab7904e27d4d18183301 -address /run/k3s/containerd/containerd.sock root 5868 1 0 Sep11 ? 00:00:27 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3e908c4d0b10df275bdef6f72fbcfa09517d11cf749236ad020364e14d77bc93 -address /run/k3s/containerd/containerd.sock root 6158 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id bec5780a5c73fa3154aa3b5ee26cdf23202db821205893e7e66ae17e6103e97b -address /run/k3s/containerd/containerd.sock root 7366 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a81b78845bdcaef710314c93e5ea0d0617f37a8929472f7b570ab90c6667f57f -address /run/k3s/containerd/containerd.sock root 97274 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id c90652c935d9e79af45b9a9ac2b4fe315e2a761a0525737fdb95c680123a164c -address /run/k3s/containerd/containerd.sock root 98309 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 431a2763636488efd7d104c228d41f4f2ecd8c06a7fc375d8977ab0d238936a8 -address /run/k3s/containerd/containerd.sock root 98493 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b480d2fa55c8cc0105ec1902887040c017f03d5b1eb6c73c24bb9d523ad9b37 -address /run/k3s/containerd/containerd.sock +``` + +### 1.2.12 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... +Permissive - Enabling Pod Security Policy can cause applications to unexpectedly fail. + +### 1.2.13 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.14 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.15 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.16 Ensure that the --secure-port argument is not set to 0 - NoteThis recommendation is obsolete and will be deleted per the consensus process (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.17 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.18 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log +Permissive. + +### 1.2.19 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 +Permissive. + +### 1.2.20 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 +Permissive. + +### 1.2.21 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 +Permissive. + +### 1.2.22 Ensure that the --request-timeout argument is set as appropriate (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, --request-timeout=300s +Permissive. + +### 1.2.23 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.24 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +### 1.2.25 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.2.29 +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.26 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +### 1.2.27 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.28 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.29 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= +Permissive - Enabling encryption changes how data can be recovered as data is encrypted. + +### 1.2.30 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. +Permissive - Enabling encryption changes how data can be recovered as data is encrypted. + +### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true +Not Applicable. + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 663 1 0 Sep11 ? 00:00:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 3021 1 0 Sep11 ? 00:00:29 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4790d392966915d995e666002c56ed4cce6dd86f305ce2ee390547a1fcbf6c82 -address /run/k3s/containerd/containerd.sock root 3035 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9d73e6a160ccde6c7c7d4ba0df3b9f696e3de2ebfc0f19a1dcbdf13aea496427 -address /run/k3s/containerd/containerd.sock root 3235 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 587f6221ee9f36c877231ada8d816a799dfda186332c33378d1eb16c72cdc87d -address /run/k3s/containerd/containerd.sock root 4435 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a74c9ec7d99785c2f2d4e6826aa80c22eb8b38249e8f99679ece00a818e9b7b3 -address /run/k3s/containerd/containerd.sock root 4985 1 0 Sep11 ? 00:00:53 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b0c9784dbe0fcbe8be10c857976e70ec84a208cb814e87b5ca085a02d434f8c -address /run/k3s/containerd/containerd.sock root 5056 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id f3231ff35f18056e74eda14907f296be15f7ea1c6ae5ab7904e27d4d18183301 -address /run/k3s/containerd/containerd.sock root 5868 1 0 Sep11 ? 00:00:27 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3e908c4d0b10df275bdef6f72fbcfa09517d11cf749236ad020364e14d77bc93 -address /run/k3s/containerd/containerd.sock root 6158 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id bec5780a5c73fa3154aa3b5ee26cdf23202db821205893e7e66ae17e6103e97b -address /run/k3s/containerd/containerd.sock root 7366 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a81b78845bdcaef710314c93e5ea0d0617f37a8929472f7b570ab90c6667f57f -address /run/k3s/containerd/containerd.sock root 97274 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id c90652c935d9e79af45b9a9ac2b4fe315e2a761a0525737fdb95c680123a164c -address /run/k3s/containerd/containerd.sock root 98309 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 431a2763636488efd7d104c228d41f4f2ecd8c06a7fc375d8977ab0d238936a8 -address /run/k3s/containerd/containerd.sock root 98493 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b480d2fa55c8cc0105ec1902887040c017f03d5b1eb6c73c24bb9d523ad9b37 -address /run/k3s/containerd/containerd.sock +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.1 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.2 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.3 +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.4 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.5 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.6 +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.7 +``` + +**Expected Result**: + +```console +'trusted-ca-file' is present +``` + +**Returned Value**: + +```console +trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +### 3.1.2 Service account token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of service account tokens. + +### 3.1.3 Bootstrap token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of bootstrap tokens. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Manual) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable - All configuration is passed in as arguments at container run time. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable. + All configuration is passed in as arguments at container run time. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +**Returned Value**: + +```console +600 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +**Returned Value**: + +```console +644 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/kubelet/config.yaml + +### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml +Not Applicable. +All configuration is passed in as arguments at container run time. + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" | grep -v grep; else echo "--anonymous-auth=false"; fi' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" | grep -v grep; else echo "--authorization-mode=Webhook"; fi' +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" | grep -v grep; else echo "--client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt"; fi' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:15 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:15Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-12-34 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=af02ecbc-1e4e-422e-8b4d-4b2aa24a9d46 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' +``` + +### 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' +``` + +**Expected Result**: + +```console +'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:15 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:15Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-12-34 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=af02ecbc-1e4e-422e-8b4d-4b2aa24a9d46 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.7 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable. + +### 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--event-qps' is present OR '--event-qps' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** Not Applicable + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service +Permissive - When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. + +### 4.2.10 Ensure that the --rotate-certificates argument is not set to false (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** pass + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable. + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'RotateKubeletServerCertificate' is present OR 'RotateKubeletServerCertificate' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.13 Ensure that a limit is set on pod PIDs (Manual) + + +**Result:** warn + +**Remediation:** +Decide on an appropriate level for this parameter and set it, +either via the --pod-max-pids command line parameter or the PodPidsLimit configuration file setting. + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--pod-max-pids' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[]?.kind=="ServiceAccount" and .subjects[]?.name=="default") or (.subjects[]?.kind=="Group" and .subjects[]?.name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[]? != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +### 5.1.9 Minimize access to create persistent volumes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to PersistentVolume objects in the cluster. + +### 5.1.10 Minimize access to the proxy sub-resource of nodes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the proxy sub-resource of node objects. + +### 5.1.11 Minimize access to the approval sub-resource of certificatesigningrequests objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the approval sub-resource of certificatesigningrequest objects. + +### 5.1.12 Minimize access to webhook configuration objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the validatingwebhookconfigurations or mutatingwebhookconfigurations objects + +### 5.1.13 Minimize access to the service account token creation (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the token sub-resource of serviceaccount objects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +### 5.2.7 Minimize the admission of root containers (Automated) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Automated) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md deleted file mode 100644 index 16614a7e520..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md +++ /dev/null @@ -1,3148 +0,0 @@ ---- -title: K3s Self-Assessment Guide - CIS Benchmark v1.7 - K8s v1.25 ---- - -This document is a companion to the [K3s Hardening Guide](../../../../pages-for-subheaders/k3s-hardening-guide.md), which provides prescriptive guidance on how to harden K3s clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 | - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.7. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -Each control in the CIS Kubernetes Benchmark was evaluated against a K3s cluster that was configured according to the accompanying hardening guide. - -Where control audits differ from the original CIS benchmark, the audit commands specific to K3s are provided for testing. - -These are the possible results for each control: - -- **Pass** - The K3s cluster passes the audit outlined in the benchmark. -- **Not Applicable** - The control is not applicable to K3s because of how it is designed to operate. The remediation section explains why. -- **Warn** - The control is manual in the CIS benchmark and it depends on the cluster's use-case or some other factor that must be determined by the cluster operator. These controls have been evaluated to ensure K3s doesn't prevent their implementation, but no further configuration or auditing of the cluster has been performed. - -This guide makes the assumption that K3s is running as a Systemd unit. Your installation may vary. Adjust the "audit" commands to fit your scenario. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Control Plane Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the -control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-apiserver.yaml - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-controller-manager.yaml - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-scheduler.yaml - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 /etc/kubernetes/manifests/etcd.yaml - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /etc/kubernetes/manifests/etcd.yaml - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 1.1.11 -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** Not Applicable - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/admin.conf - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 scheduler - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root scheduler - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 controllermanager - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/controller.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root controllermanager - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown -R root:root /etc/kubernetes/pki/ - -**Audit:** - -```bash -find /var/lib/rancher/k3s/server/tls | xargs stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) - - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - -**Audit:** - -```bash -stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) - - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /etc/kubernetes/pki/*.key - -**Audit:** - -```bash -stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep containerd | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' is present OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 519 1 0 22:09 ? 00:00:00 /usr/bin/containerd root 801 1 0 22:09 ? 00:00:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 3864 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id d00174abbc275f6bb85c7f0be1d3154b9c91982a10b9dba6b5cb280f4d4c531d -address /run/k3s/containerd/containerd.sock root 4105 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7c2b546b4d2380bcb51278661f34cff94fad2ba06978e13f8f1b92dafcc89d43 -address /run/k3s/containerd/containerd.sock root 4206 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 68d8a55ff4663985be004608dbf78b0362f5522e18490c81d4c8dc9963de1556 -address /run/k3s/containerd/containerd.sock root 5374 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id ca0ae9e0b37dfd7b1ce05f72e1bc5a1be8f5cb08f2b4543081536de3bdbc925d -address /run/k3s/containerd/containerd.sock root 5443 1 0 22:31 ? 00:00:01 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3ea3c1cdbbd5adb8efd5c67a46aadd0fca9918dc0ad1f7cafe38b83171e3dc1b -address /run/k3s/containerd/containerd.sock root 7130 1 0 22:32 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4d838297d35a31003106ac5989c3547433985bb2964b47baad12cee6e375645e -address /run/k3s/containerd/containerd.sock root 7639 1 0 22:32 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 341cb9bcd8486aa2f1acb8e1ae51baebd630ac6ed266643266c34d677f61c7d0 -address /run/k3s/containerd/containerd.sock root 10308 1 0 23:17 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id c534fbee8e0d06fd9b29bf8fc70a138975c6b18db25f1faf2615677dfdb4199e -address /run/k3s/containerd/containerd.sock root 11370 1 0 23:18 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4ff4b8776dac7a35b83616d341dbe4d5a689ac7fb9b8eee8db5978e3968380ea -address /run/k3s/containerd/containerd.sock root 13736 13723 2 23:21 ? 00:00:10 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 16022 1 0 23:29 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9027256349086e458119478e5e00384b1b76fbf5e6dbee23699f596a88d9f2bc -address /run/k3s/containerd/containerd.sock root 16159 1 0 23:29 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 929bf369fc5881654f4c1925624151ddb7cea51073267b8d213d966ba45406f3 -address /run/k3s/containerd/containerd.sock -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'AlwaysPullImages' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'ServiceAccount' -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-path' -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxage' -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxbackup' -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxsize' -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.23 Ensure that the --request-timeout argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameter as appropriate and if needed. -For example, --request-timeout=300s - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'request-timeout' -``` - -**Expected Result**: - -```console -'--request-timeout' is not present OR '--request-timeout' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-lookup' -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.25 Ensure that the --service-account-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-key-file' -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 1.2.29 -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'encryption-provider-config' -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.31 Ensure that encryption providers are appropriately configured (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit:** - -```bash -grep aescbc /path/to/encryption-config.json -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) - - -**Result:** warn - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'bind-address' -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.1 -``` - -**Expected Result**: - -```console -'cert-file' is present AND 'key-file' is present -``` - -**Returned Value**: - -```console -cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key -``` - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and set the below parameter. ---client-cert-auth="true" - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.2 -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -client-cert-auth: true -``` - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.3 -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.4 -``` - -**Expected Result**: - -```console -'cert-file' is present AND 'key-file' is present -``` - -**Returned Value**: - -```console -cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key -``` - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and set the below parameter. ---peer-client-cert-auth=true - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.5 -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -client-cert-auth: true -``` - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.6 -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.7 -``` - -**Expected Result**: - -```console -'trusted-ca-file' is present -``` - -**Returned Value**: - -```console -trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Manual) - - -**Result:** warn - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig -``` - -**Expected Result**: - -```console -'permissions' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/agent/kubelet.kubeconfig -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt -``` - -**Expected Result**: - -```console -'644' is equal to '644' OR '640' is present OR '600' is present OR '444' is present OR '440' is present OR '400' is present OR '000' is present -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" | grep -v grep; else echo "--anonymous-auth=false"; fi' -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" | grep -v grep; else echo "--authorization-mode=Webhook"; fi' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" | grep -v grep; else echo "--client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt"; fi' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'protect-kernel-defaults' -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' -``` - -**Expected Result**: - -```console -'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Manual) - - -**Result:** warn - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** warn - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) - - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md new file mode 100644 index 00000000000..903e42374f3 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md @@ -0,0 +1,515 @@ +--- +title: RKE 加固指南 +--- + + + + + +本文档提供了针对生产环境的 RKE 集群进行加固的具体指导，以便在使用 Rancher 部署之前进行配置。它概述了满足信息安全中心（Center for Information Security, CIS）Kubernetes benchmark controls 所需的配置和控制。 + +:::note +这份加固指南描述了如何确保你集群中的节点安全。我们建议你在安装 Kubernetes 之前遵循本指南。 +::: + +此加固指南适用于 RKE 集群，并与以下版本的 CIS Kubernetes Benchmark、Kubernetes 和 Rancher 相关联： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|------------------------------| +| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 至 v1.26 | + +:::note +- 在 Benchmark v1.24 及更高版本中，检查 id `4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automated)` 可能会失败，因为 `/etc/kubernetes/ssl/kube-ca.pem` 默认设置为 644。 +- 在 Benchmark v1.7 中，不再需要 `--protect-kernel-defaults` (`4.2.6`) 参数，并已被 CIS 删除。 +::: + +有关如何评估加固的 RKE 集群与官方 CIS benchmark 的更多细节，请参考特定 Kubernetes 和 CIS benchmark 版本的 RKE 自我评估指南。 + +## 主机级别要求 + +### 配置 Kernel 运行时参数 + +建议对群集中的所有节点类型使用以下 `sysctl` 配置。在 `/etc/sysctl.d/90-kubelet.conf` 中设置以下参数： + +```ini +vm.overcommit_memory=1 +vm.panic_on_oom=0 +kernel.panic=10 +kernel.panic_on_oops=1 +``` + +运行 `sysctl -p /etc/sysctl.d/90-kubelet.conf` 以启用设置。 + +### 配置 `etcd` 用户和组 + +在安装 RKE 之前，需要设置 **etcd** 服务的用户帐户和组。 + +#### 创建 `etcd` 用户和组 + +要创建 **etcd** 用户和组，请运行以下控制台命令。 +下面的命令示例中使用 `52034` 作为 **uid** 和 **gid** 。 +任何有效且未使用的 **uid** 或 **gid** 都可以代替 `52034`。 + +```bash +groupadd --gid 52034 etcd +useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd --shell /usr/sbin/nologin +``` + +在通过集群配置文件 `config.yml` 部署RKE时，请更新 `etcd` 用户的 `uid` 和 `gid`： + +```yaml +services: + etcd: + gid: 52034 + uid: 52034 +``` + +## Kubernetes 运行时要求 + +### 配置 `default` Service Account + +#### 设置 `automountServiceAccountToken` 为 `false` 用于 `default` service accounts + +Kubernetes 提供了一个 default service account，供集群工作负载使用，其中没有为 pod 分配特定的 service account。 +如果需要从 pod 访问 Kubernetes API，则应为该 pod 创建特定的 service account，并向该 service account 授予权限。 +应配置 default service account，使其不提供 service account 令牌，并且不应具有任何明确的权限分配。 + +对于标准 RKE 安装上的每个命名空间（包括 `default` 和 `kube-system`），`default` service account 必须包含以下值： + +```yaml +automountServiceAccountToken: false +``` + +将以下配置保存到名为 `account_update.yaml` 的文件中。 + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: default +automountServiceAccountToken: false +``` + +创建一个名为 `account_update.yaml` 的 bash 脚本文件。 +确保执行 `chmod +x account_update.sh` 命令，以赋予脚本执行权限。 + +```bash +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" +done +``` + +执行此脚本将 `account_update.yaml` 配置应用到所有命名空间中的 `default` service account。 + +### 配置网络策略 + +#### 确保所有命名空间都定义了网络策略 + +在同一个 Kubernetes 集群上运行不同的应用程序会带来风险，即某个受感染的应用程序可能会攻击相邻的应用程序。为确保容器只与其预期通信的容器进行通信，网络分段至关重要。网络策略规定了哪些 Pod 可以互相通信，以及与其他网络终端通信的方式。 + +网络策略是命名空间范围的。当在特定命名空间引入网络策略时，所有未被策略允许的流量将被拒绝。然而，如果在命名空间中没有网络策略，那么所有流量将被允许进入和离开该命名空间中的 Pod。要强制执行网络策略，必须启用容器网络接口（container network interface, CNI）插件。本指南使用 [Canal](https://github.com/projectcalico/canal) 来提供策略执行。有关 CNI 提供程序的其他信息可以在[这里](https://www.suse.com/c/rancher_blog/comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/)找到。 + +一旦在集群上启用了 CNI 提供程序，就可以应用默认的网络策略。下面提供了一个 **permissive** 的示例供参考。如果你希望允许匹配某个命名空间中所有 Pod 的所有入站和出站流量（即使添加了策略导致某些 Pod 被视为”隔离”），你可以创建一个明确允许该命名空间中所有流量的策略。请将以下配置保存为 `default-allow-all.yaml`。有关网络策略的其他[文档](https://kubernetes.io/docs/concepts/services-networking/network-policies/)可以在 Kubernetes 站点上找到。 + +:::caution +此网络策略只是一个示例，不建议用于生产用途。 +::: + +```yaml +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-allow-all +spec: + podSelector: {} + ingress: + - {} + egress: + - {} + policyTypes: + - Ingress + - Egress +``` + +创建一个名为 `apply_networkPolicy_to_all_ns.sh`的 Bash 脚本文件。 + +确保运行 `chmod +x apply_networkPolicy_to_all_ns.sh` 命令，以赋予脚本执行权限。 + +```bash +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + kubectl apply -f default-allow-all.yaml -n ${namespace} +done +``` + +执行此脚本以将 `default-allow-all.yaml` 配置和 **permissive** 的 `NetworkPolicy` 应用于所有命名空间。 + +## 已知限制 + +- 当注册自定义节点仅提供公共 IP 时，Rancher **exec shell** 和 **查看 pod 日志** 在加固设置中**不起作用**。 此功能需要在注册自定义节点时提供私有 IP。 +- 当根据 Rancher [提供](../../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md)的 Pod 安全策略 (Pod Security Policies, PSP) 将 `default_pod_security_policy_template_id:` 设置为 `restricted` 或 `restricted-noroot` 时，Rancher 会在 `default` service accounts 上创建 `RoleBindings` 和 `ClusterRoleBindings`。CIS 检查 5.1.5 要求除了默认角色之外，`default` service accounts 不应绑定其他角色或集群角色。此外，`default` service accounts 应配置为不提供服务账户令牌，也不具有任何明确的权限分配。 + +## 加固的 RKE `cluster.yml` 配置参考 + +参考的 `cluster.yml` 文件是由 RKE CLI 使用的，它提供了实现 RKE 加固安装所需的配置。 +RKE [文档](https://rancher.com/docs/rke/latest/en/installation/)提供了有关配置项的更多详细信息。这里参考的 `cluster.yml` 不包括必需的 `nodes` 指令，因为它取决于你的环境。在 RKE 中有关节点配置的文档可以在[这里](https://rancher.com/docs/rke/latest/en/config-options/nodes/)找到。 + +示例 `cluster.yml` 配置文件中包含了一个 Admission Configuration 策略，在 `services.kube-api.admission_configuration` 字段中指定。这个[示例](../../psa-restricted-exemptions.md)策略包含了命名空间的豁免规则，这对于在Rancher中正确运行导入的RKE集群非常必要，类似于Rancher预定义的 [`rancher-restricted`](../../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md) 策略。 + +如果你希望使用 RKE 的默认 `restricted` 策略，则将 `services.kube-api.admission_configuration` 字段留空，并将 `services.pod_security_configuration` 设置为 `restricted`。你可以在 [RKE 文档](https://rke.docs.rancher.com/config-options/services/pod-security-admission)中找到更多信息。 + + + + +:::note +如果你打算将一个 RKE 集群导入到 Rancher 中，请参考此[文档](../../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md)以了解如何配置 PSA 以豁免 Rancher 系统命名空间。 +::: + +```yaml +# 如果你打算在离线环境部署 Kubernetes， +# 请查阅文档以了解如何配置自定义的 RKE 镜像。 +nodes: [] +kubernetes_version: # 定义 RKE 版本 +services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + secrets_encryption_config: + enabled: true + audit_log: + enabled: true + event_rate_limit: + enabled: true + # 如果你在 `admission_configuration` 中设置了自定义策略， + # 请将 `pod_security_configuration` 字段留空。 + # 否则，将其设置为 `restricted` 以使用 RKE 预定义的受限策略， + # 并删除 `admission_configuration` 字段中的所有内容。 + # + # pod_security_configuration: restricted + # + admission_configuration: + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1 + kind: PodSecurityConfiguration + defaults: + enforce: "restricted" + enforce-version: "latest" + audit: "restricted" + audit-version: "latest" + warn: "restricted" + warn-version: "latest" + exemptions: + usernames: [] + runtimeClasses: [] + namespaces: [calico-apiserver, + calico-system, + cattle-alerting, + cattle-csp-adapter-system, + cattle-elemental-system, + cattle-epinio-system, + cattle-externalip-system, + cattle-fleet-local-system, + cattle-fleet-system, + cattle-gatekeeper-system, + cattle-global-data, + cattle-global-nt, + cattle-impersonation-system, + cattle-istio, + cattle-istio-system, + cattle-logging, + cattle-logging-system, + cattle-monitoring-system, + cattle-neuvector-system, + cattle-prometheus, + cattle-provisioning-capi-system, + cattle-resources-system, + cattle-sriov-system, + cattle-system, + cattle-ui-plugin-system, + cattle-windows-gmsa-system, + cert-manager, + cis-operator-system, + fleet-default, + ingress-nginx, + istio-system, + kube-node-lease, + kube-public, + kube-system, + longhorn-system, + rancher-alerting-drivers, + security-scan, + tigera-operator] + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + generate_serving_certificate: true +addons: | + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: default-allow-all + spec: + podSelector: {} + ingress: + - {} + egress: + - {} + policyTypes: + - Ingress + - Egress + --- + apiVersion: v1 + kind: ServiceAccount + metadata: + name: default + automountServiceAccountToken: false +``` + + + + +```yaml +# 如果你打算在离线环境部署 Kubernetes， +# 请查阅文档以了解如何配置自定义的 RKE 镜像。 +nodes: [] +kubernetes_version: # 定义 RKE 版本 +services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + secrets_encryption_config: + enabled: true + audit_log: + enabled: true + event_rate_limit: + enabled: true + pod_security_policy: true + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: true + generate_serving_certificate: true +addons: | + # Upstream Kubernetes restricted PSP policy + # https://github.com/kubernetes/website/blob/564baf15c102412522e9c8fc6ef2b5ff5b6e766c/content/en/examples/policy/restricted-psp.yaml + apiVersion: policy/v1beta1 + kind: PodSecurityPolicy + metadata: + name: restricted-noroot + spec: + privileged: false + # Required to prevent escalations to root. + allowPrivilegeEscalation: false + requiredDropCapabilities: + - ALL + # Allow core volume types. + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'downwardAPI' + # Assume that ephemeral CSI drivers & persistentVolumes set up by the cluster admin are safe to use. + - 'csi' + - 'persistentVolumeClaim' + - 'ephemeral' + hostNetwork: false + hostIPC: false + hostPID: false + runAsUser: + # Require the container to run without root privileges. + rule: 'MustRunAsNonRoot' + seLinux: + # This policy assumes the nodes are using AppArmor rather than SELinux. + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + # Forbid adding the root group. + - min: 1 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + # Forbid adding the root group. + - min: 1 + max: 65535 + readOnlyRootFilesystem: false + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: psp:restricted-noroot + rules: + - apiGroups: + - extensions + resourceNames: + - restricted-noroot + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: psp:restricted-noroot + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted-noroot + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: default-allow-all + spec: + podSelector: {} + ingress: + - {} + egress: + - {} + policyTypes: + - Ingress + - Egress + --- + apiVersion: v1 + kind: ServiceAccount + metadata: + name: default + automountServiceAccountToken: false +``` + + + + +## 加固后的 RKE 集群模板配置参考 + +参考的 RKE 集群模板提供了实现 Kubernetes 加固安装所需的最低配置。RKE 模板用于提供 Kubernetes 并定义 Rancher 设置。有关安装 RKE 及其模板详情的其他信息，请参考 Rancher [文档](../../../../getting-started/installation-and-upgrade/installation-and-upgrade.md) 。 + + + + +```yaml +# +# 集群配置 +# +default_pod_security_admission_configuration_template_name: rancher-restricted +enable_network_policy: true +local_cluster_auth_endpoint: + enabled: true +name: # 定义集群名称 + +# +# Rancher 配置 +# +rancher_kubernetes_engine_config: + addon_job_timeout: 45 + authentication: + strategy: x509|webhook + kubernetes_version: # 定义 RKE 版本 + services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + audit_log: + enabled: true + event_rate_limit: + enabled: true + pod_security_policy: false + secrets_encryption_config: + enabled: true + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + generate_serving_certificate: true + scheduler: + extra_args: + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + + + + +```yaml +# +# 集群配置 +# +default_pod_security_policy_template_id: restricted-noroot +enable_network_policy: true +local_cluster_auth_endpoint: + enabled: true +name: # 定义集群名称 + +# +# Rancher 配置 +# +rancher_kubernetes_engine_config: + addon_job_timeout: 45 + authentication: + strategy: x509|webhook + kubernetes_version: # 定义 RKE 版本 + services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + audit_log: + enabled: true + event_rate_limit: + enabled: true + pod_security_policy: true + secrets_encryption_config: + enabled: true + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + generate_serving_certificate: true + scheduler: + extra_args: + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + + + + +## 结论 + +如果你按照本指南操作，由 Rancher 提供的 RKE 自定义集群将配置为通过 CIS Kubernetes Benchmark 测试。你可以查看我们的 RKE 自我评估指南，了解我们是如何验证每个 benchmarks 的，并且你可以在你的集群上执行相同的操作。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md index 34bb63f89f0..7dfb1204916 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md @@ -1,30 +1,34 @@ --- -title: RKE Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.23 +title: RKE 自我评估指南 - CIS Benchmark v1.23 - K8s v1.23 --- -This document is a companion to the [RKE Hardening Guide](../../../../pages-for-subheaders/rke1-hardening-guide.md), which provides prescriptive guidance on how to harden RKE clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. + + + -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: +本文档是 [RKE 加固指南](rke1-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 -| Rancher Version | CIS Benchmark Version | Kubernetes Version | +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | |-----------------|-----------------------|--------------------| | Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 -This document is for Rancher operators, security teams, auditors and decision makers. +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.23 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 -## Testing Methodology +## 测试方法 -Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. +Rancher 和 RKE 通过 Docker 容器安装 Kubernetes 服务。配置是通过初始化时传递给容器的参数定义的，而不是通过配置文件。 -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 :::note -This guide only covers `automated` (previously called `scored`) tests. +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md deleted file mode 100644 index 25108b848b5..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md +++ /dev/null @@ -1,3084 +0,0 @@ ---- -title: RKE Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.24 ---- - -This document is a companion to the [RKE Hardening Guide](../../../../pages-for-subheaders/rke1-hardening-guide.md), which provides prescriptive guidance on how to harden RKE clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.24 | - -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Control Plane Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 permissions=600 -``` - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root -``` - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit:** - -```bash -stat -c %a /node/var/lib/etcd -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** Not Applicable - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chown -R root:root /etc/kubernetes/pki/ - -**Audit Script:** `check_files_owner_in_dir.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the owner is set to root:root for -# the given directory and all the files in it -# -# inputs: -# $1 = /full/path/to/directory -# -# outputs: -# true/false - -INPUT_DIR=$1 - -if [[ "${INPUT_DIR}" == "" ]]; then - echo "false" - exit -fi - -if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then - echo "false" - exit -fi - -statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) -while read -r statInfoLine; do - f=$(echo ${statInfoLine} | cut -d' ' -f1) - p=$(echo ${statInfoLine} | cut -d' ' -f2) - - if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then - if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "root:root" ]]; then - echo "false" - exit - fi - fi -done <<< "${statInfoLines}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - -**Audit Script:** `check_files_permissions.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_permissions.sh '/node/etc/kubernetes/ssl/!(*key).pem' -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /etc/kubernetes/ssl/*key.pem - -**Audit Script:** `check_files_permissions.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_permissions.sh '/node/etc/kubernetes/ssl/*key.pem' -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.31 Ensure that encryption providers are appropriately configured (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit Script:** `check_encryption_provider_config.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to check the encrption provider config is set to aesbc -# -# outputs: -# true/false - -# TODO: Figure out the file location from the kube-apiserver commandline args -ENCRYPTION_CONFIG_FILE="/node/etc/kubernetes/ssl/encryption.yaml" - -if [[ ! -f "${ENCRYPTION_CONFIG_FILE}" ]]; then - echo "false" - exit -fi - -for provider in "$@" -do - if grep "$provider" "${ENCRYPTION_CONFIG_FILE}"; then - echo "true" - exit - fi -done - -echo "false" -exit - -``` - -**Audit Execution:** - -```bash -./check_encryption_provider_config.sh aescbc -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -- aescbc: true -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 5056 5035 2 18:52 ? 00:00:05 kube-controller-manager --cloud-provider= --terminated-pod-gc-threshold=1000 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --node-monitor-grace-period=40s --allow-untagged-cloud=true --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --configure-cloud-routes=false --leader-elect=true --profiling=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --pod-eviction-timeout=5m0s --v=2 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --enable-hostpath-provisioner=false --use-service-account-credentials=true -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5056 5035 2 18:52 ? 00:00:05 kube-controller-manager --cloud-provider= --terminated-pod-gc-threshold=1000 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --node-monitor-grace-period=40s --allow-untagged-cloud=true --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --configure-cloud-routes=false --leader-elect=true --profiling=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --pod-eviction-timeout=5m0s --v=2 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --enable-hostpath-provisioner=false --use-service-account-credentials=true -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 5056 5035 2 18:52 ? 00:00:05 kube-controller-manager --cloud-provider= --terminated-pod-gc-threshold=1000 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --node-monitor-grace-period=40s --allow-untagged-cloud=true --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --configure-cloud-routes=false --leader-elect=true --profiling=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --pod-eviction-timeout=5m0s --v=2 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --enable-hostpath-provisioner=false --use-service-account-credentials=true -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 5056 5035 2 18:52 ? 00:00:05 kube-controller-manager --cloud-provider= --terminated-pod-gc-threshold=1000 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --node-monitor-grace-period=40s --allow-untagged-cloud=true --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --configure-cloud-routes=false --leader-elect=true --profiling=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --pod-eviction-timeout=5m0s --v=2 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --enable-hostpath-provisioner=false --use-service-account-credentials=true -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 5056 5035 2 18:52 ? 00:00:05 kube-controller-manager --cloud-provider= --terminated-pod-gc-threshold=1000 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --node-monitor-grace-period=40s --allow-untagged-cloud=true --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --configure-cloud-routes=false --leader-elect=true --profiling=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --pod-eviction-timeout=5m0s --v=2 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --enable-hostpath-provisioner=false --use-service-account-credentials=true -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -Cluster provisioned by RKE handles certificate rotation directly through RKE. - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 5056 5035 2 18:52 ? 00:00:05 kube-controller-manager --cloud-provider= --terminated-pod-gc-threshold=1000 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --node-monitor-grace-period=40s --allow-untagged-cloud=true --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --configure-cloud-routes=false --leader-elect=true --profiling=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --pod-eviction-timeout=5m0s --v=2 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --enable-hostpath-provisioner=false --use-service-account-credentials=true -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5209 5190 0 18:53 ? 00:00:01 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 5209 5190 0 18:53 ? 00:00:01 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--cert-file' is present AND '--key-file' is present -``` - -**Returned Value**: - -```console -root 4720 4700 3 18:52 ? 00:00:08 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --advertise-client-urls=https://172.31.26.105:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --listen-peer-urls=https://172.31.26.105:2380 --initial-cluster=etcd-ip-172-31-26-105=https://172.31.26.105:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --heartbeat-interval=500 --data-dir=/var/lib/rancher/etcd/ --listen-client-urls=https://172.31.26.105:2379 --name=etcd-ip-172-31-26-105 --initial-cluster-state=new --client-cert-auth=true --election-timeout=5000 --initial-advertise-peer-urls=https://172.31.26.105:2380 root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false root 18033 17938 2 18:57 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -root 4720 4700 3 18:52 ? 00:00:08 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --advertise-client-urls=https://172.31.26.105:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --listen-peer-urls=https://172.31.26.105:2380 --initial-cluster=etcd-ip-172-31-26-105=https://172.31.26.105:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --heartbeat-interval=500 --data-dir=/var/lib/rancher/etcd/ --listen-client-urls=https://172.31.26.105:2379 --name=etcd-ip-172-31-26-105 --initial-cluster-state=new --client-cert-auth=true --election-timeout=5000 --initial-advertise-peer-urls=https://172.31.26.105:2380 root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false root 18033 17938 4 18:57 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-26-105 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem ETCDCTL_ENDPOINTS=https://172.31.26.105:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/root -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-cert-file' is present AND '--peer-key-file' is present -``` - -**Returned Value**: - -```console -root 4720 4700 3 18:52 ? 00:00:08 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --advertise-client-urls=https://172.31.26.105:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --listen-peer-urls=https://172.31.26.105:2380 --initial-cluster=etcd-ip-172-31-26-105=https://172.31.26.105:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --heartbeat-interval=500 --data-dir=/var/lib/rancher/etcd/ --listen-client-urls=https://172.31.26.105:2379 --name=etcd-ip-172-31-26-105 --initial-cluster-state=new --client-cert-auth=true --election-timeout=5000 --initial-advertise-peer-urls=https://172.31.26.105:2380 root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false root 18033 17938 3 18:57 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -root 4720 4700 3 18:52 ? 00:00:08 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --advertise-client-urls=https://172.31.26.105:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --listen-peer-urls=https://172.31.26.105:2380 --initial-cluster=etcd-ip-172-31-26-105=https://172.31.26.105:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --heartbeat-interval=500 --data-dir=/var/lib/rancher/etcd/ --listen-client-urls=https://172.31.26.105:2379 --name=etcd-ip-172-31-26-105 --initial-cluster-state=new --client-cert-auth=true --election-timeout=5000 --initial-advertise-peer-urls=https://172.31.26.105:2380 root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false root 18033 17938 2 18:57 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-26-105 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem ETCDCTL_ENDPOINTS=https://172.31.26.105:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/root -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--trusted-ca-file' is present -``` - -**Returned Value**: - -```console -root 4720 4700 3 18:52 ? 00:00:08 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --advertise-client-urls=https://172.31.26.105:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --listen-peer-urls=https://172.31.26.105:2380 --initial-cluster=etcd-ip-172-31-26-105=https://172.31.26.105:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --heartbeat-interval=500 --data-dir=/var/lib/rancher/etcd/ --listen-client-urls=https://172.31.26.105:2379 --name=etcd-ip-172-31-26-105 --initial-cluster-state=new --client-cert-auth=true --election-timeout=5000 --initial-advertise-peer-urls=https://172.31.26.105:2380 root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false root 18033 17938 2 18:57 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-policy-file' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:43 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-node.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/kubernetes/ssl/kubecfg-kube-node.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit:** - -```bash -stat -c permissions=%a /node/etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit:** - -```bash -stat -c %U:%G /node/etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--streaming-connection-idle-timeout' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--event-qps' is equal to '0' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE handles certificate rotation directly through RKE. - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -**Audit Script:** `check_for_default_sa.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` - -**Audit Execution:** - -```bash -./check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): serviceaccounts is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "serviceaccounts" in API group "" at the cluster scope Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-fleet-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-impersonation-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cis-operator-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-node-lease" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-public" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "local" true -``` - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** warn - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) - - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -**Audit Script:** `check_for_network_policies.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in $(kubectl get namespaces --all-namespaces -o json | jq -r '.items[].metadata.name'); do - policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') - if [[ ${policy_count} -eq 0 ]]; then - echo "false" - exit - fi -done - -echo "true" - -``` - -**Audit Execution:** - -```bash -./check_for_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is present -``` - -**Returned Value**: - -```console -true -``` - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Automated) - - -**Result:** pass - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. - -**Audit Script:** `check_for_default_ns.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count=$(kubectl get all -n default -o json | jq .items[] | jq -r 'select((.metadata.name!="kubernetes"))' | jq .metadata.name | wc -l) -if [[ ${count} -gt 0 ]]; then - echo "false" - exit -fi - -echo "true" - - -``` - -**Audit Execution:** - -```bash -./check_for_default_ns.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): replicationcontrollers is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "replicationcontrollers" in API group "" in the namespace "default" Error from server (Forbidden): services is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "services" in API group "" in the namespace "default" Error from server (Forbidden): daemonsets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "daemonsets" in API group "apps" in the namespace "default" Error from server (Forbidden): deployments.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "deployments" in API group "apps" in the namespace "default" Error from server (Forbidden): replicasets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "replicasets" in API group "apps" in the namespace "default" Error from server (Forbidden): statefulsets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "statefulsets" in API group "apps" in the namespace "default" Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default" Error from server (Forbidden): cronjobs.batch is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "cronjobs" in API group "batch" in the namespace "default" Error from server (Forbidden): jobs.batch is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "jobs" in API group "batch" in the namespace "default" true -``` - diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md deleted file mode 100644 index 0cec041061f..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md +++ /dev/null @@ -1,3085 +0,0 @@ ---- -title: RKE Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.25 ---- - -This document is a companion to the [RKE Hardening Guide](../../../../pages-for-subheaders/rke1-hardening-guide.md), which provides prescriptive guidance on how to harden RKE clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.25 | - -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Control Plane Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 permissions=600 -``` - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root -``` - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit:** - -```bash -stat -c %a /node/var/lib/etcd -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** Not Applicable - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chown -R root:root /etc/kubernetes/pki/ - -**Audit Script:** `check_files_owner_in_dir.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the owner is set to root:root for -# the given directory and all the files in it -# -# inputs: -# $1 = /full/path/to/directory -# -# outputs: -# true/false - -INPUT_DIR=$1 - -if [[ "${INPUT_DIR}" == "" ]]; then - echo "false" - exit -fi - -if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then - echo "false" - exit -fi - -statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) -while read -r statInfoLine; do - f=$(echo ${statInfoLine} | cut -d' ' -f1) - p=$(echo ${statInfoLine} | cut -d' ' -f2) - - if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then - if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "root:root" ]]; then - echo "false" - exit - fi - fi -done <<< "${statInfoLines}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - -**Audit Script:** `check_files_permissions.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_permissions.sh '/node/etc/kubernetes/ssl/!(*key).pem' -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /etc/kubernetes/ssl/*key.pem - -**Audit Script:** `check_files_permissions.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_permissions.sh '/node/etc/kubernetes/ssl/*key.pem' -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.31 Ensure that encryption providers are appropriately configured (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit Script:** `check_encryption_provider_config.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to check the encrption provider config is set to aesbc -# -# outputs: -# true/false - -# TODO: Figure out the file location from the kube-apiserver commandline args -ENCRYPTION_CONFIG_FILE="/node/etc/kubernetes/ssl/encryption.yaml" - -if [[ ! -f "${ENCRYPTION_CONFIG_FILE}" ]]; then - echo "false" - exit -fi - -for provider in "$@" -do - if grep "$provider" "${ENCRYPTION_CONFIG_FILE}"; then - echo "true" - exit - fi -done - -echo "false" -exit - -``` - -**Audit Execution:** - -```bash -./check_encryption_provider_config.sh aescbc -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -- aescbc: true -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -Cluster provisioned by RKE handles certificate rotation directly through RKE. - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5671 5649 0 22:01 ? 00:00:01 kube-scheduler --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 5671 5649 0 22:01 ? 00:00:01 kube-scheduler --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--cert-file' is present AND '--key-file' is present -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 5 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 4 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-51 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem ETCDCTL_ENDPOINTS=https://172.31.31.51:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-cert-file' is present AND '--peer-key-file' is present -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 2 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 3 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-51 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem ETCDCTL_ENDPOINTS=https://172.31.31.51:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--trusted-ca-file' is present -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 2 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-policy-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:34 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-node.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/kubernetes/ssl/kubecfg-kube-node.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit:** - -```bash -stat -c permissions=%a /node/etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit:** - -```bash -stat -c %U:%G /node/etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--streaming-connection-idle-timeout' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--event-qps' is equal to '0' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE handles certificate rotation directly through RKE. - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -**Audit Script:** `check_for_default_sa.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` - -**Audit Execution:** - -```bash -./check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): serviceaccounts is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "serviceaccounts" in API group "" at the cluster scope Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-fleet-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-impersonation-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cis-operator-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-node-lease" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-public" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "local" true -``` - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** warn - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) - - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -**Audit Script:** `check_for_network_policies.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in $(kubectl get namespaces --all-namespaces -o json | jq -r '.items[].metadata.name'); do - policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') - if [[ ${policy_count} -eq 0 ]]; then - echo "false" - exit - fi -done - -echo "true" - -``` - -**Audit Execution:** - -```bash -./check_for_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is present -``` - -**Returned Value**: - -```console -true -``` - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Automated) - - -**Result:** pass - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. - -**Audit Script:** `check_for_default_ns.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count=$(kubectl get all -n default -o json | jq .items[] | jq -r 'select((.metadata.name!="kubernetes"))' | jq .metadata.name | wc -l) -if [[ ${count} -gt 0 ]]; then - echo "false" - exit -fi - -echo "true" - - -``` - -**Audit Execution:** - -```bash -./check_for_default_ns.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): replicationcontrollers is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "replicationcontrollers" in API group "" in the namespace "default" Error from server (Forbidden): services is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "services" in API group "" in the namespace "default" Error from server (Forbidden): daemonsets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "daemonsets" in API group "apps" in the namespace "default" Error from server (Forbidden): deployments.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "deployments" in API group "apps" in the namespace "default" Error from server (Forbidden): replicasets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "replicasets" in API group "apps" in the namespace "default" Error from server (Forbidden): statefulsets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "statefulsets" in API group "apps" in the namespace "default" Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default" Error from server (Forbidden): cronjobs.batch is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "cronjobs" in API group "batch" in the namespace "default" Error from server (Forbidden): jobs.batch is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "jobs" in API group "batch" in the namespace "default" true -``` - diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md new file mode 100644 index 00000000000..2246a13f6d0 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md @@ -0,0 +1,3046 @@ +--- +title: RKE 自我评估指南 - CIS Benchmark v1.24 - K8s v1.24 +--- + + + + + +本文档是 [RKE 加固指南](rke1-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | + +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.24 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +Rancher 和 RKE 通过 Docker 容器安装 Kubernetes 服务。配置是通过初始化时传递给容器的参数定义的，而不是通过配置文件。 + +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. +All configuration is passed in as arguments at container run time. + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. +All configuration is passed in as arguments at container run time. + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=644 +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root root:root +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c %a /node/var/lib/etcd +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** Not Applicable + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit Script:** `check_files_owner_in_dir.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to ensure the owner is set to root:root for +# the given directory and all the files in it +# +# inputs: +# $1 = /full/path/to/directory +# +# outputs: +# true/false + +INPUT_DIR=$1 + +if [[ "${INPUT_DIR}" == "" ]]; then + echo "false" + exit +fi + +if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then + echo "false" + exit +fi + +statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) +while read -r statInfoLine; do + f=$(echo ${statInfoLine} | cut -d' ' -f1) + p=$(echo ${statInfoLine} | cut -d' ' -f2) + + if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then + if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "root:root" ]]; then + echo "false" + exit + fi + fi +done <<< "${statInfoLines}" + + +echo "true" +exit + +``` + +**Audit Execution:** + +```bash +./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' -exec chmod -R 600 {} + + +**Audit:** + +```bash +find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' | xargs stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=644 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/ssl/*key.pem + +**Audit Script:** `check_files_permissions.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to ensure the file permissions are set to 644 or +# more restrictive for all files in a given directory or a wildcard +# selection of files +# +# inputs: +# $1 = /full/path/to/directory or /path/to/fileswithpattern +# ex: !(*key).pem +# +# $2 (optional) = permission (ex: 600) +# +# outputs: +# true/false + +# Turn on "extended glob" for use of '!' in wildcard +shopt -s extglob + +# Turn off history to avoid surprises when using '!' +set -H + +USER_INPUT=$1 + +if [[ "${USER_INPUT}" == "" ]]; then + echo "false" + exit +fi + + +if [[ -d ${USER_INPUT} ]]; then + PATTERN="${USER_INPUT}/*" +else + PATTERN="${USER_INPUT}" +fi + +PERMISSION="" +if [[ "$2" != "" ]]; then + PERMISSION=$2 +fi + +FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) + +while read -r fileInfo; do + p=$(echo ${fileInfo} | cut -d' ' -f2) + + if [[ "${PERMISSION}" != "" ]]; then + if [[ "$p" != "${PERMISSION}" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then + echo "false" + exit + fi + fi +done <<< "${FILES_PERMISSIONS}" + + +echo "true" +exit + +``` + +**Audit Execution:** + +```bash +./check_files_permissions.sh '/node/etc/kubernetes/ssl/*key.pem' +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --kubelet-https parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-https' is present OR '--kubelet-https' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.18 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxsize' is greater or equal to 100 +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.31 Ensure that encryption providers are appropriately configured (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit Script:** `check_encryption_provider_config.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to check the encrption provider config is set to aesbc +# +# outputs: +# true/false + +# TODO: Figure out the file location from the kube-apiserver commandline args +ENCRYPTION_CONFIG_FILE="/node/etc/kubernetes/ssl/encryption.yaml" + +if [[ ! -f "${ENCRYPTION_CONFIG_FILE}" ]]; then + echo "false" + exit +fi + +for provider in "$@" +do + if grep "$provider" "${ENCRYPTION_CONFIG_FILE}"; then + echo "true" + exit + fi +done + +echo "false" +exit + +``` + +**Audit Execution:** + +```bash +./check_encryption_provider_config.sh aescbc +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +- aescbc: true +``` + +### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +Cluster provisioned by RKE handles certificate rotation directly through RKE. + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 3859 3838 0 Sep11 ? 00:03:44 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 3859 3838 0 Sep11 ? 00:03:44 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--cert-file' is present AND '--key-file' is present +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:08 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 5 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:08 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 4 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-7-100 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-cert-file' is present AND '--peer-key-file' is present +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 2 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 2 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-7-100 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--trusted-ca-file' is present +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 2 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:10 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. +All configuration is passed in as arguments at container run time. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. +All configuration is passed in as arguments at container run time. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit:** + +```bash +stat -c permissions=%a /node/etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit:** + +```bash +stat -c %U:%G /node/etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/kubelet/config.yaml + +Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml + +Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--streaming-connection-idle-timeout' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--protect-kernel-defaults' is equal to 'true' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors + +### 4.2.9 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--event-qps' is equal to '0' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +Clusters provisioned by RKE handles certificate rotation directly through RKE. + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Automated) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[]?.kind=="ServiceAccount" and .subjects[]?.name=="default") or (.subjects[]?.kind=="Group" and .subjects[]?.name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[]? != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=1 +``` + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=1 +``` + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=1 +``` + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=1 +``` + +### 5.2.7 Minimize the admission of root containers (Manual) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +**Audit Script:** `check_for_network_policies.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +for namespace in $(kubectl get namespaces --all-namespaces -o json | jq -r '.items[].metadata.name'); do + policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') + if [[ ${policy_count} -eq 0 ]]; then + echo "false" + exit + fi +done + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_network_policies.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Automated) + + +**Result:** pass + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + +**Audit Script:** `check_for_default_ns.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count=$(kubectl get all -n default -o json | jq .items[] | jq -r 'select((.metadata.name!="kubernetes"))' | jq .metadata.name | wc -l) +if [[ ${count} -gt 0 ]]; then + echo "false" + exit +fi + +echo "true" + + +``` + +**Audit Execution:** + +```bash +./check_for_default_ns.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md new file mode 100644 index 00000000000..5e5a6b1eed2 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md @@ -0,0 +1,2862 @@ +--- +title: RKE 自我评估指南 - CIS Benchmark v1.7 - K8s v1.25/v1.26/v1.27 +--- + + + + + +本文档是 [RKE 加固指南](rke1-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25/v1.26/v1.27 | + +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.7 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +Rancher 和 RKE 通过 Docker 容器安装 Kubernetes 服务。配置是通过初始化时传递给容器的参数定义的，而不是通过配置文件。 + +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-apiserver.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-controller-manager.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-scheduler.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /etc/kubernetes/manifests/etcd.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /etc/kubernetes/manifests/etcd.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +**Audit:** + +```bash +ps -ef | grep kubelet | grep -- --cni-conf-dir | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +**Audit:** + +```bash +ps -ef | grep kubelet | grep -- --cni-conf-dir | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c %a /node/var/lib/etcd +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +**Audit:** + +```bash +stat -c %U:%G /node/var/lib/etcd +``` + +**Expected Result**: + +```console +'etcd:etcd' is present +``` + +**Returned Value**: + +```console +etcd:etcd +``` + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/admin.conf +Not Applicable - Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf +Not Applicable - Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 scheduler +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root scheduler +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 controllermanager +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root controllermanager +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit Script:** `check_files_owner_in_dir.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to ensure the owner is set to root:root for +# the given directory and all the files in it +# +# inputs: +# $1 = /full/path/to/directory +# +# outputs: +# true/false + +INPUT_DIR=$1 + +if [[ "${INPUT_DIR}" == "" ]]; then + echo "false" + exit +fi + +if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then + echo "false" + exit +fi + +statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) +while read -r statInfoLine; do + f=$(echo ${statInfoLine} | cut -d' ' -f1) + p=$(echo ${statInfoLine} | cut -d' ' -f2) + + if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then + if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "root:root" ]]; then + echo "false" + exit + fi + fi +done <<< "${statInfoLines}" + + +echo "true" +exit + +``` + +**Audit Execution:** + +```bash +./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' -exec chmod -R 600 {} + + +**Audit:** + +```bash +find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' | xargs stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=644 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +find /node/etc/kubernetes/ssl/ -name '*key.pem' -exec chmod -R 600 {} + + +**Audit:** + +```bash +find /node/etc/kubernetes/ssl/ -name '*key.pem' | xargs stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= +When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. + +### 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.7 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.9 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.12 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.13 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.14 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.15 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.16 Ensure that the --secure-port argument is not set to 0 - NoteThis recommendation is obsolete and will be deleted per the consensus process (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.17 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.18 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.19 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.20 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.21 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxsize' is greater or equal to 100 +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.22 Ensure that the --request-timeout argument is set as appropriate (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, --request-timeout=300s + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +### 1.2.23 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.24 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.25 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.26 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.27 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.28 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.29 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.30 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit:** + +```bash +ENCRYPTION_PROVIDER_CONFIG=$(ps -ef | grep kube-apiserver | grep -- --encryption-provider-config | sed 's%.*encryption-provider-config[= ]\([^ ]*\).*%\1%') if test -e $ENCRYPTION_PROVIDER_CONFIG; then grep -A1 'providers:' $ENCRYPTION_PROVIDER_CONFIG | tail -n1 | grep -o "[A-Za-z]*" | sed 's/^/provider=/'; fi +``` + +**Expected Result**: + +```console +'provider' is present +``` + +### 1.2.31 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true +Cluster provisioned by RKE handles certificate rotation directly through RKE. + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 4339 4318 0 Sep11 ? 00:03:28 kube-scheduler --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --profiling=false --v=2 --leader-elect=true +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 4339 4318 0 Sep11 ? 00:03:28 kube-scheduler --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --profiling=false --v=2 --leader-elect=true +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--cert-file' is present AND '--key-file' is present +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 2 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 1 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-4-224 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-cert-file' is present AND '--peer-key-file' is present +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 2 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 1 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-4-224 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--trusted-ca-file' is present +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 1 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +### 3.1.2 Service account token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of service account tokens. + +### 3.1.3 Bootstrap token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of bootstrap tokens. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable - Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. +All configuration is passed in as arguments at container run time. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable - Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. + All configuration is passed in as arguments at container run time. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit:** + +```bash +stat -c permissions=%a /node/etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit:** + +```bash +stat -c %U:%G /node/etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/kubelet/config.yaml +Not Applicable - Clusters provisioned by RKE do not require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml +Not Applicable - Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--streaming-connection-idle-timeout' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.7 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable - Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors + +### 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--event-qps' is greater or equal to 0 OR '--event-qps' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.10 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable - Clusters provisioned by RKE handles certificate rotation directly through RKE. + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +### 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.13 Ensure that a limit is set on pod PIDs (Manual) + + +**Result:** warn + +**Remediation:** +Decide on an appropriate level for this parameter and set it, +either via the --pod-max-pids command line parameter or the PodPidsLimit configuration file setting. + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--pod-max-pids' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[]?.kind=="ServiceAccount" and .subjects[]?.name=="default") or (.subjects[]?.kind=="Group" and .subjects[]?.name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[]? != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +### 5.1.9 Minimize access to create persistent volumes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to PersistentVolume objects in the cluster. + +### 5.1.10 Minimize access to the proxy sub-resource of nodes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the proxy sub-resource of node objects. + +### 5.1.11 Minimize access to the approval sub-resource of certificatesigningrequests objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the approval sub-resource of certificatesigningrequest objects. + +### 5.1.12 Minimize access to webhook configuration objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the validatingwebhookconfigurations or mutatingwebhookconfigurations objects + +### 5.1.13 Minimize access to the service account token creation (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the token sub-resource of serviceaccount objects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +### 5.2.7 Minimize the admission of root containers (Manual) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md deleted file mode 100644 index 084568cc402..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md +++ /dev/null @@ -1,3085 +0,0 @@ ---- -title: RKE Self-Assessment Guide - CIS Benchmark v1.7 - K8s v1.25 ---- - -This document is a companion to the [RKE Hardening Guide](../../../../pages-for-subheaders/rke1-hardening-guide.md), which provides prescriptive guidance on how to harden RKE clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 | - -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.7. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Control Plane Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 permissions=600 -``` - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root -``` - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit:** - -```bash -stat -c %a /node/var/lib/etcd -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** Not Applicable - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chown -R root:root /etc/kubernetes/pki/ - -**Audit Script:** `check_files_owner_in_dir.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the owner is set to root:root for -# the given directory and all the files in it -# -# inputs: -# $1 = /full/path/to/directory -# -# outputs: -# true/false - -INPUT_DIR=$1 - -if [[ "${INPUT_DIR}" == "" ]]; then - echo "false" - exit -fi - -if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then - echo "false" - exit -fi - -statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) -while read -r statInfoLine; do - f=$(echo ${statInfoLine} | cut -d' ' -f1) - p=$(echo ${statInfoLine} | cut -d' ' -f2) - - if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then - if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "root:root" ]]; then - echo "false" - exit - fi - fi -done <<< "${statInfoLines}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - -**Audit Script:** `check_files_permissions.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_permissions.sh '/node/etc/kubernetes/ssl/!(*key).pem' -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /etc/kubernetes/ssl/*key.pem - -**Audit Script:** `check_files_permissions.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_permissions.sh '/node/etc/kubernetes/ssl/*key.pem' -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.31 Ensure that encryption providers are appropriately configured (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit Script:** `check_encryption_provider_config.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to check the encrption provider config is set to aesbc -# -# outputs: -# true/false - -# TODO: Figure out the file location from the kube-apiserver commandline args -ENCRYPTION_CONFIG_FILE="/node/etc/kubernetes/ssl/encryption.yaml" - -if [[ ! -f "${ENCRYPTION_CONFIG_FILE}" ]]; then - echo "false" - exit -fi - -for provider in "$@" -do - if grep "$provider" "${ENCRYPTION_CONFIG_FILE}"; then - echo "true" - exit - fi -done - -echo "false" -exit - -``` - -**Audit Execution:** - -```bash -./check_encryption_provider_config.sh aescbc -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -- aescbc: true -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -Cluster provisioned by RKE handles certificate rotation directly through RKE. - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5671 5649 0 22:01 ? 00:00:01 kube-scheduler --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 5671 5649 0 22:01 ? 00:00:01 kube-scheduler --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--cert-file' is present AND '--key-file' is present -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 5 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 4 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-51 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem ETCDCTL_ENDPOINTS=https://172.31.31.51:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-cert-file' is present AND '--peer-key-file' is present -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 2 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 3 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-51 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem ETCDCTL_ENDPOINTS=https://172.31.31.51:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--trusted-ca-file' is present -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 2 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-policy-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:34 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-node.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/kubernetes/ssl/kubecfg-kube-node.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit:** - -```bash -stat -c permissions=%a /node/etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit:** - -```bash -stat -c %U:%G /node/etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--streaming-connection-idle-timeout' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--event-qps' is equal to '0' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE handles certificate rotation directly through RKE. - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -**Audit Script:** `check_for_default_sa.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` - -**Audit Execution:** - -```bash -./check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): serviceaccounts is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "serviceaccounts" in API group "" at the cluster scope Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-fleet-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-impersonation-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cis-operator-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-node-lease" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-public" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "local" true -``` - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** warn - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) - - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -**Audit Script:** `check_for_network_policies.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in $(kubectl get namespaces --all-namespaces -o json | jq -r '.items[].metadata.name'); do - policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') - if [[ ${policy_count} -eq 0 ]]; then - echo "false" - exit - fi -done - -echo "true" - -``` - -**Audit Execution:** - -```bash -./check_for_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is present -``` - -**Returned Value**: - -```console -true -``` - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Automated) - - -**Result:** pass - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. - -**Audit Script:** `check_for_default_ns.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count=$(kubectl get all -n default -o json | jq .items[] | jq -r 'select((.metadata.name!="kubernetes"))' | jq .metadata.name | wc -l) -if [[ ${count} -gt 0 ]]; then - echo "false" - exit -fi - -echo "true" - - -``` - -**Audit Execution:** - -```bash -./check_for_default_ns.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): replicationcontrollers is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "replicationcontrollers" in API group "" in the namespace "default" Error from server (Forbidden): services is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "services" in API group "" in the namespace "default" Error from server (Forbidden): daemonsets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "daemonsets" in API group "apps" in the namespace "default" Error from server (Forbidden): deployments.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "deployments" in API group "apps" in the namespace "default" Error from server (Forbidden): replicasets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "replicasets" in API group "apps" in the namespace "default" Error from server (Forbidden): statefulsets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "statefulsets" in API group "apps" in the namespace "default" Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default" Error from server (Forbidden): cronjobs.batch is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "cronjobs" in API group "batch" in the namespace "default" Error from server (Forbidden): jobs.batch is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "jobs" in API group "batch" in the namespace "default" true -``` - diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md new file mode 100644 index 00000000000..431d87bb1ed --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md @@ -0,0 +1,273 @@ +--- +title: RKE2 加固指南 +--- + + + + + +本文档提供了针对生产环境的 RKE2 集群进行加固的具体指导，以便在使用 Rancher 部署之前进行配置。它概述了满足信息安全中心（Center for Information Security, CIS）Kubernetes benchmark controls 制所需的配置和控制。 + +:::note +这份加固指南描述了如何确保你集群中的节点安全。我们建议你在安装 Kubernetes 之前遵循本指南。 +::: + +此加固指南适用于 RKE2 集群，并与以下版本的 CIS Kubernetes Benchmark、Kubernetes 和 Rancher 相关联： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|------------------------------| +| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 至 v1.26 | + +:::note +- 在 Benchmark v1.24 及更高版本中，由于新的文件权限要求（600 而不是 644），一些检查 ID 可能会失败。受影响的检查 ID 包括：`1.1.1`, `1.1.3`, `1.1.5`, `1.1.7`, `1.1.13`, `1.1.15`, `1.1.17`, `4.1.3`, `4.1.5` 和 `4.1.9`。 +- 在 Benchmark v1.7 中，不再需要 `--protect-kernel-defaults` (4.2.6) 参数，并已被 CIS 删除。 +::: + +有关如何评估加固的 RKE2 集群与官方 CIS benchmark 的更多细节，请参考特定 Kubernetes 和 CIS benchmark 版本的 RKE2 自我评估指南。 + +RKE2 在不需要修改的情况下通过了许多 Kubernetes CIS controls，因为它默认应用了几个安全缓解措施。然而，有一些值得注意的例外情况，需要手动干预才能完全符合 CIS Benchmark 要求： + +1. RKE2 不会修改主机操作系统。因此，作为运维人员，你必须进行一些主机级别的修改。 +2. 某些 CIS controls 对于网络策略和 Pod 安全标准（或 RKE2 v1.25 之前的 Pod 安全策略 (PSP)）将限制集群的功能。你必须选择让 RKE2 为你配置这些功能。为了确保满足这些要求，可以启动 RKE2 并设置 profile 标志为 `cis-1.23`（适用于 v1.25 及更新版本）或 `cis-1.6`（适用于 v1.24 及更早版本）。 + +## 主机级别要求 + +主机级要求有两个方面：内核参数和 etcd 进程/目录配置。这些在本节中进行了概述。 + +### 确保 `protect-kernel-defaults` 已经设置 + + + + +自 CIS benchmark 1.7 开始，不再需要`protect-kernel-defaults`。 + + + + +这是一个 kubelet 标志，如果所需的内核参数未设置或设置为与 kubelet 的默认值不同的值，将导致 kubelet 退出。 + +可以在 Rancher 的集群配置中设置 `protect-kernel-defaults` 标志。 + +```yaml +spec: + rkeConfig: + machineSelectorConfig: + - config: + protect-kernel-defaults: true +``` + + + + +### 设置内核参数 + +建议为集群中的所有节点类型设置以下 `sysctl` 配置。在 `/etc/sysctl.d/90-kubelet.conf` 中设置以下参数： + +```ini +vm.panic_on_oom=0 +vm.overcommit_memory=1 +kernel.panic=10 +kernel.panic_on_oops=1 +``` + +运行 `sudo sysctl -p /etc/sysctl.d/90-kubelet.conf` 以启用设置。 + +### 确保 etcd 配置正确 + +CIS Benchmark 要求 etcd 数据目录由 `etcd` 用户和组拥有。这意味着 etcd 进程必须以主机级别的 `etcd` 用户身份运行。为了实现这一点，在使用有效的 `cis-1.xx` 配置文件启动 RKE2 时，RKE2 会采取以下几个步骤： + +1. 检查主机上是否存在 `etcd` 用户和组。如果不存在，则显示错误并退出。 +2. 使用 `etcd` 作为用户和组所有者创建 etcd 的数据目录。 +3. 通过适当设置 etcd 静态 Pod 的 `SecurityContext`，确保 etcd 进程以 `etcd` 用户和组的身份运行。 + +为满足上述要求，你必须执行以下操作： + +#### 创建 etcd 用户 + +在某些 Linux 发行版中，`useradd` 命令不会创建组。下面包含了 `-U` 标志来解决这个问题。这个标志告诉 `useradd` 创建一个与用户同名的组。 + +```bash +sudo useradd -r -c "etcd user" -s /sbin/nologin -M etcd -U +``` + +## Kubernetes 运行时要求 + +通过 CIS Benchmark 测试的运行时要求主要集中在 Pod 安全、网络策略和内核参数上。当使用有效的 `cis-1.xx` 配置文件时，大部分都会被 RKE2 自动处理，但需要一些额外的运维人员干预是。本节概述了这些内容。 + +### Pod 安全 + +RKE2 始终以一定程度的 Pod 安全性运行。 + + + + +在 v1.25 及更高版本中，[Pod 安全准入(PSAs)](https://kubernetes.io/docs/concepts/security/pod-security-admission/)用于保证 pod 安全。 + +以下是确保加固 RKE2 通过 Rancher 中提供的 CIS v1.23 加固配置文件 `rke2-cis-1.7-hardened` 所需的最低配置。 + +```yaml +spec: + defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted + rkeConfig: + machineSelectorConfig: + - config: + profile: cis-1.23 +``` + +当同时设置了 `defaultPodSecurityAdmissionConfigurationTemplateName` 和 `profile` 标志时，Rancher 和 RKE2 会执行以下操作： + +1. 检查是否已满足主机级要求。如果未满足，RKE2 将以致命错误退出，并描述未满足的要求。 +2. 应用允许群集传递关联 controls 的网络策略。 +3. 配置 Pod 安全准入控制器（PSA）使用 PSA 配置模板 `rancher-restricted`，以在所有命名空间中强制执行受限模式，除了模板豁免列表中的命名空间。 + 这些命名空间被豁免，以允许系统 Pod 在没有限制的情况下运行，这是集群正常运行所必需的。 + +:::note +如果你打算将一个 RKE 集群导入到 Rancher 中，请参考[文档](../../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md)了解如何配置 PSA 以豁免 Rancher system 命名空间。 +::: + + + + + +在 Kubernetes v1.24 及更早版本中，`PodSecurityPolicy` 准入控制器始终是启用的。 + +以下是确保 RKE2 加固以通过 Rancher 中提供的 CIS v1.23 加固配置文件 `rke2-cis-1.23-hardened` 所需的最低配置。 + +:::note +在下面的示例中，配置文件设置为`cis-1.6`，这是在上游 RKE2 中定义的值，但集群实际上配置为传递 CIS v1.23 加固配置文件 +::: + +```yaml +spec: + defaultPodSecurityPolicyTemplateName: restricted-noroot + rkeConfig: + machineSelectorConfig: + - config: + profile: cis-1.6 +``` + + +当同时设置了 `defaultPodSecurityPolicyTemplateName` 和 `profile` 标志时，Rancher 和 RKE2 会执行以下操作： + +1. 检查是否已满足主机级要求。如果未满足，RKE2 将以致命错误退出，并描述未满足的要求。 +2. 应用网络策略，以确保集群通过相关的 controls 要求。 +3. 配置运行时 Pod 安全策略，以确保集群通过相关的 controls 要求。 + + + + +:::note +Kubernetes control plane 组件以及关键的附加组件，如 CNI、DNS 和 Ingress，都作为 `kube-system` 命名空间中的 Pod 运行。因此，这个命名空间的限制政策较少，从而使这些组件能够正常运行。 +::: + +### 网络策略 + +当使用有效的 `cis-1.xx` 配置文件运行时，RKE2 将设置适当的 `NetworkPolicies`，以满足 Kubernetes 内置命名空间的 CIS Benchmark 要求。这些命名空间包括：`kube-system`、`kube-public`、`kube-node-lease` 和 `default`。 + +所使用的 `NetworkPolicy` 仅允许同一命名空间内的 Pod 相互通信。值得注意的例外是允许 DNS 请求进行解析。 + +:::note +运维人员必须像管理其他命名空间一样管理额外创建的命名空间的网络策略。 +::: + +### 配置 `default` service account + +**将 `default` service accountsSet 的 `automountServiceAccountToken` 设置为 `false`** + +Kubernetes 提供了一个 `default` service account，用于集群工作负载，在 pod 没有分配特定 service account 时使用。如果需要从 pod 访问 Kubernetes API，则应为该 pod 创建一个特定的 service account，并授予该 service account 权限。`default` service account 应配置为不提供 service account 令牌，并且不具有任何明确的权限分配。 + +对于标准的 RKE2 安装中的每个命名空间，包括 `default` 和 `kube-system`，`default` service account 必须包含此值： + +```yaml +automountServiceAccountToken: false +``` + +对于由集群操作员创建的命名空间，可以使用以下脚本和配置文件来配置 `default` service account。 + +以下配置必须保存到一个名为 `account_update.yaml` 的文件中。 + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: default +automountServiceAccountToken: false +``` + +创建一个名为 `account_update.sh` 的 bash 脚本文件。确保运行 `sudo chmod +x account_update.sh` 命令，以便脚本具有执行权限。 + +```bash +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + echo -n "Patching namespace $namespace - " + kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" +done +``` + +执行此脚本以将 `account_update.yaml` 配置应用到所有命名空间中的 `default` service account。 + +### API server 审计配置 + +CIS 要求 1.2.19 至 1.2.22 与为 API server 配置审计日志有关。当 RKE2 在设置配置文件标志的情况下启动时，它将自动在 API server 中配置加固的 `--audit-log-` 参数以通过这些 CIS 检查。 + +RKE2 的默认审计策略被配置为不记录 API server 中的请求。这样做是为了允许集群操作员灵活地定制符合其审计要求和需求的审计策略，因为这些策略是针对每个用户的环境和政策而特定的。 + +当使用 `profile` 标志启动 RKE2 时，RKE2 会创建一个默认的审计策略。该策略定义在 `/etc/rancher/rke2/audit-policy.yaml` 中。 + +```yaml +apiVersion: audit.k8s.io/v1 +kind: Policy +metadata: + creationTimestamp: null +rules: +- level: None +``` + +## 加固的 RKE2 模板配置参考 + +参考模板配置用于在 Rancher 中创建加固的 RKE2 自定义集群。该参考不包括其他必需的**集群配置**指令，这些指令会根据你的环境而有所不同。 + + + + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: # 定义集群名称 +spec: + defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted + kubernetesVersion: # 定义 RKE2 版本 + rkeConfig: + machineSelectorConfig: + - config: + profile: cis-1.23 +``` + + + + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: # 定义集群名称 +spec: + defaultPodSecurityPolicyTemplateName: restricted-noroot + kubernetesVersion: # 定义 RKE2 版本 + rkeConfig: + machineSelectorConfig: + - config: + profile: cis-1.6 + protect-kernel-defaults: true +``` + + + + +## 结论 + +如果你按照本指南操作，由 Rancher 提供的 RKE2 自定义集群将配置为通过 CIS Kubernetes Benchmark 测试。你可以查看我们的 RKE2 自我评估指南，了解我们是如何验证每个 benchmarks 的，并且你可以在你的集群上执行相同的操作。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md index 83cac7d6e59..05ec2b4aef5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md @@ -1,30 +1,34 @@ --- -title: RKE2 Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.23 +title: RKE2 自我评估指南 - CIS Benchmark v1.23 - K8s v1.23 --- -This document is a companion to the [RKE2 Hardening Guide](../../../../pages-for-subheaders/rke2-hardening-guide.md), which provides prescriptive guidance on how to harden RKE2 clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. + + + -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: +本文档是 [RKE2 加固指南](rke2-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE2 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 -| Rancher Version | CIS Benchmark Version | Kubernetes Version | +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | |-----------------|-----------------------|--------------------| | Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE2 install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE2 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 -This document is for Rancher operators, security teams, auditors and decision makers. +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.23 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 -## Testing Methodology +## 测试方法 -RKE2 launches control plane components as static pods, managed by the kubelet, and uses containerd as the container runtime. Configuration is defined by arguments passed to the container at the time of initialization or via configuration file. +RKE2 将 control plane 组件作为静态 Pod 启动，由 kubelet 管理，并使用 containerd 作为容器运行时。配置是由初始化时或通过配置文件传递给容器的参数定义的。 -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE2 nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE2 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 :::note -This guide only covers `automated` (previously called `scored`) tests. +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md deleted file mode 100644 index ffb7755d7ad..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md +++ /dev/null @@ -1,3196 +0,0 @@ ---- -title: RKE2 Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.24 ---- - -This document is a companion to the [RKE2 Hardening Guide](../../../../pages-for-subheaders/rke2-hardening-guide.md), which provides prescriptive guidance on how to harden RKE2 clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.24 | - -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE2 install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -RKE2 launches control plane components as static pods, managed by the kubelet, and uses containerd as the container runtime. Configuration is defined by arguments passed to the container at the time of initialization or via configuration file. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE2 nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Master Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the -control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %a /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -644 -``` - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' -``` - -**Expected Result**: - -```console -'permissions' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 permissions=644 -``` - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root -``` - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/db/etcd -``` - -**Expected Result**: - -```console -permissions has permissions 700, expected 700 or more restrictive -``` - -**Returned Value**: - -```console -permissions=700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/db/etcd -``` - -**Expected Result**: - -```console -'etcd:etcd' is present -``` - -**Returned Value**: - -```console -etcd:etcd -``` - -### 1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 600 /etc/kubernetes/admin.conf - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/admin.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/admin.conf - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/admin.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 scheduler - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root scheduler - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 controllermanager - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/controller.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /var/lib/rancher/rke2/server/cred/controller.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/controller.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown -R root:root /var/lib/rancher/rke2/server/tls/ - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/tls -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 644 /var/lib/rancher/rke2/server/tls/*.crt - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.crt -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /var/lib/rancher/rke2/server/tls/*.key - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.key -``` - -**Expected Result**: - -```console -'permissions' is equal to '600' -``` - -**Returned Value**: - -```console -permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'AlwaysPullImages' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -### 1.2.33 Ensure that encryption providers are appropriately configured (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit:** - -```bash -/bin/sh -c 'if grep aescbc /var/lib/rancher/rke2/server/cred/encryption-config.json; then echo 0; fi' -``` - -**Expected Result**: - -```console -'0' is present -``` - -**Returned Value**: - -```console -{"kind":"EncryptionConfiguration","apiVersion":"apiserver.config.k8s.io/v1","resources":[{"resources":["secrets"],"providers":[{"aescbc":{"keys":[{"name":"aescbckey","secret":"LdLeqCPN/HfmgnUBXVPkDtUfeOPHwcQzDiLYG3nzFI4="}]}},{"identity":{}}]}]} 0 -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3652 3523 0 23:45 ? 00:00:03 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 3652 3523 0 23:45 ? 00:00:03 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-102 ETCD_UNSUPPORTED_ARCH= FILE_HASH=755e8914c854015cfb808b8456f808644387876672add1f9e19564c58f754595 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=78088791fd2612fad13ce760ef91a0ee HOME=/ -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-102 ETCD_UNSUPPORTED_ARCH= FILE_HASH=755e8914c854015cfb808b8456f808644387876672add1f9e19564c58f754595 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=78088791fd2612fad13ce760ef91a0ee HOME=/ -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Audit Config:** - -```bash -cat /var/lib/rancher/rke2/server/db/etcd/config -``` - -**Expected Result**: - -```console -'ETCD_TRUSTED_CA_FILE' is present OR '{.peer-transport-security.trusted-ca-file}' is equal to '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-102 ETCD_UNSUPPORTED_ARCH= FILE_HASH=755e8914c854015cfb808b8456f808644387876672add1f9e19564c58f754595 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=78088791fd2612fad13ce760ef91a0ee HOME=/ -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep | grep -o audit-policy-file -``` - -**Expected Result**: - -```console -'audit-policy-file' is equal to 'audit-policy-file' -``` - -**Returned Value**: - -```console -audit-policy-file -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive OR '/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is present OR '/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/kubelet.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit Script:** `check_cafile_permissions.sh` - -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -CAFILE=/node$CAFILE -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi - -``` - -**Audit Execution:** - -```bash -./check_cafile_permissions.sh -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit Script:** `check_cafile_ownership.sh` - -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -CAFILE=/node$CAFILE -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi - -``` - -**Audit Execution:** - -```bash -./check_cafile_ownership.sh -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /etc/rancher/rke2/rke2.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/rancher/rke2/rke2.yaml; then stat -c permissions=%a /etc/rancher/rke2/rke2.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /etc/rancher/rke2/rke2.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/rancher/rke2/rke2.yaml; then stat -c %U:%G /etc/rancher/rke2/rke2.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3221 3096 3 23:44 ? 00:00:19 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-102 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=499dafa1-6719-45d4-a0c6-66109f61b011 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3221 3096 3 23:44 ? 00:00:19 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-102 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=499dafa1-6719-45d4-a0c6-66109f61b011 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3221 3096 3 23:44 ? 00:00:19 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-102 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=499dafa1-6719-45d4-a0c6-66109f61b011 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3221 3096 3 23:44 ? 00:00:19 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-102 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=499dafa1-6719-45d4-a0c6-66109f61b011 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJIZXg2aVo0Q09SYmlTd1pyYmRHa2ZIaHNKb2RZaGRScDFjNklkMngKS05ITHZzZEdxeEc0V2xySVp6WTEwTDlLWDJCVk9Senp0RUZ4cDc3MTlFTnVnNDZqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU1pFWGkwbDNsS3dXS1U3S0x6CkgwZG5aTzVEOWpBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCK2NQUUhLdXBJc05zU3BLVEg2NHJibFpKYTIyZ0oKdlh4TmNxb0dQSEdhUlFJZ1JJdU1hcFVKOHVyVEdxblZGckFldy81aFdOZFdRUjNyN3FXQmlrYnk0aEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRpZ0F3SUJBZ0lJUnIzVlVFbTY4Yll3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU5UQXpOekFlRncweU16QXlNall5TXpRek5UZGFGdzB5TkRBeQpNall5TXpRek5UZGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUTE5SWdKZVJyUldOVGUKZC9UUWcwNWdNcWFMSU9oUU1HSUZKRytTSFJUTVl1RkxBUHhydGozV1g3T3hIRDFDaE5nM2p6VnJKbjcwYitpUApIRzJRNS9XTW8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVViNGJ3aEFWMitabXd0MWlaN2pKMGVDTytGRWd3Q2dZSUtvWkl6ajBFQXdJRFJ3QXcKUkFJZ2ZjNHJWdWJwUWVRMmxiVUJPNkxhK0hKNWVHbUFqSTRHMTV6K1pEZmZqQ1lDSUZGVkFTd1ZzVFRsUXlvMQpicmpsdnJzVW9rRmNxN3RIYjJxZk51dTV2aTB1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxXTnMKYVdWdWRDMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTFqYkdsbGJuUXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLTzZGUG4xd3dqbWFYdzc1T1l1blI1QytMbkpiRlpNYlgyWDRVR0kKMWFKK0tXb0tMZExmRnlZbVdvbHoyV3IzUnBhNGVxRXYySTg1bndINWp1NDZnNXlqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnZodkNFQlhiNW1iQzNXSm51Ck1uUjRJNzRVU0RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCeWR6VWRaMm5IZU56UHY2RkwzYUdabXMrZ2VuTmoKLzNaditGR1dhZi9nK3dJaEFNMnVCVE9maW9vQUptRXBpSi92WUFrQmRMdkdyWEdOY2R6UzdJRzRJR3BXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUQ4MXVnTGVmd0EvekE2U2l0V3czeDNiQWRHQ3A2eXJqYmNSRE14eFgwd3FvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTmZTSUNYa2EwVmpVM25mMDBJTk9ZREttaXlEb1VEQmlCU1J2a2gwVXpHTGhTd0Q4YTdZOQoxbCt6c1J3OVFvVFlONDgxYXlaKzlHL29qeHh0a09mMWpBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3221 3096 3 23:44 ? 00:00:19 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-102 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=499dafa1-6719-45d4-a0c6-66109f61b011 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.makeIPTablesUtilChains}' is present OR '{.makeIPTablesUtilChains}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJIZXg2aVo0Q09SYmlTd1pyYmRHa2ZIaHNKb2RZaGRScDFjNklkMngKS05ITHZzZEdxeEc0V2xySVp6WTEwTDlLWDJCVk9Senp0RUZ4cDc3MTlFTnVnNDZqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU1pFWGkwbDNsS3dXS1U3S0x6CkgwZG5aTzVEOWpBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCK2NQUUhLdXBJc05zU3BLVEg2NHJibFpKYTIyZ0oKdlh4TmNxb0dQSEdhUlFJZ1JJdU1hcFVKOHVyVEdxblZGckFldy81aFdOZFdRUjNyN3FXQmlrYnk0aEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRpZ0F3SUJBZ0lJUnIzVlVFbTY4Yll3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU5UQXpOekFlRncweU16QXlNall5TXpRek5UZGFGdzB5TkRBeQpNall5TXpRek5UZGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUTE5SWdKZVJyUldOVGUKZC9UUWcwNWdNcWFMSU9oUU1HSUZKRytTSFJUTVl1RkxBUHhydGozV1g3T3hIRDFDaE5nM2p6VnJKbjcwYitpUApIRzJRNS9XTW8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVViNGJ3aEFWMitabXd0MWlaN2pKMGVDTytGRWd3Q2dZSUtvWkl6ajBFQXdJRFJ3QXcKUkFJZ2ZjNHJWdWJwUWVRMmxiVUJPNkxhK0hKNWVHbUFqSTRHMTV6K1pEZmZqQ1lDSUZGVkFTd1ZzVFRsUXlvMQpicmpsdnJzVW9rRmNxN3RIYjJxZk51dTV2aTB1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxXTnMKYVdWdWRDMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTFqYkdsbGJuUXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLTzZGUG4xd3dqbWFYdzc1T1l1blI1QytMbkpiRlpNYlgyWDRVR0kKMWFKK0tXb0tMZExmRnlZbVdvbHoyV3IzUnBhNGVxRXYySTg1bndINWp1NDZnNXlqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnZodkNFQlhiNW1iQzNXSm51Ck1uUjRJNzRVU0RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCeWR6VWRaMm5IZU56UHY2RkwzYUdabXMrZ2VuTmoKLzNaditGR1dhZi9nK3dJaEFNMnVCVE9maW9vQUptRXBpSi92WUFrQmRMdkdyWEdOY2R6UzdJRzRJR3BXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUQ4MXVnTGVmd0EvekE2U2l0V3czeDNiQWRHQ3A2eXJqYmNSRE14eFgwd3FvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTmZTSUNYa2EwVmpVM25mMDBJTk9ZREttaXlEb1VEQmlCU1J2a2gwVXpHTGhTd0Q4YTdZOQoxbCt6c1J3OVFvVFlONDgxYXlaKzlHL29qeHh0a09mMWpBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.eventRecordQPS}' is present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJIZXg2aVo0Q09SYmlTd1pyYmRHa2ZIaHNKb2RZaGRScDFjNklkMngKS05ITHZzZEdxeEc0V2xySVp6WTEwTDlLWDJCVk9Senp0RUZ4cDc3MTlFTnVnNDZqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU1pFWGkwbDNsS3dXS1U3S0x6CkgwZG5aTzVEOWpBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCK2NQUUhLdXBJc05zU3BLVEg2NHJibFpKYTIyZ0oKdlh4TmNxb0dQSEdhUlFJZ1JJdU1hcFVKOHVyVEdxblZGckFldy81aFdOZFdRUjNyN3FXQmlrYnk0aEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRpZ0F3SUJBZ0lJUnIzVlVFbTY4Yll3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU5UQXpOekFlRncweU16QXlNall5TXpRek5UZGFGdzB5TkRBeQpNall5TXpRek5UZGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUTE5SWdKZVJyUldOVGUKZC9UUWcwNWdNcWFMSU9oUU1HSUZKRytTSFJUTVl1RkxBUHhydGozV1g3T3hIRDFDaE5nM2p6VnJKbjcwYitpUApIRzJRNS9XTW8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVViNGJ3aEFWMitabXd0MWlaN2pKMGVDTytGRWd3Q2dZSUtvWkl6ajBFQXdJRFJ3QXcKUkFJZ2ZjNHJWdWJwUWVRMmxiVUJPNkxhK0hKNWVHbUFqSTRHMTV6K1pEZmZqQ1lDSUZGVkFTd1ZzVFRsUXlvMQpicmpsdnJzVW9rRmNxN3RIYjJxZk51dTV2aTB1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxXTnMKYVdWdWRDMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTFqYkdsbGJuUXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLTzZGUG4xd3dqbWFYdzc1T1l1blI1QytMbkpiRlpNYlgyWDRVR0kKMWFKK0tXb0tMZExmRnlZbVdvbHoyV3IzUnBhNGVxRXYySTg1bndINWp1NDZnNXlqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnZodkNFQlhiNW1iQzNXSm51Ck1uUjRJNzRVU0RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCeWR6VWRaMm5IZU56UHY2RkwzYUdabXMrZ2VuTmoKLzNaditGR1dhZi9nK3dJaEFNMnVCVE9maW9vQUptRXBpSi92WUFrQmRMdkdyWEdOY2R6UzdJRzRJR3BXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUQ4MXVnTGVmd0EvekE2U2l0V3czeDNiQWRHQ3A2eXJqYmNSRE14eFgwd3FvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTmZTSUNYa2EwVmpVM25mMDBJTk9ZREttaXlEb1VEQmlCU1J2a2gwVXpHTGhTd0Q4YTdZOQoxbCt6c1J3OVFvVFlONDgxYXlaKzlHL29qeHh0a09mMWpBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3221 3096 3 23:44 ? 00:00:19 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-102 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=499dafa1-6719-45d4-a0c6-66109f61b011 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJIZXg2aVo0Q09SYmlTd1pyYmRHa2ZIaHNKb2RZaGRScDFjNklkMngKS05ITHZzZEdxeEc0V2xySVp6WTEwTDlLWDJCVk9Senp0RUZ4cDc3MTlFTnVnNDZqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU1pFWGkwbDNsS3dXS1U3S0x6CkgwZG5aTzVEOWpBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCK2NQUUhLdXBJc05zU3BLVEg2NHJibFpKYTIyZ0oKdlh4TmNxb0dQSEdhUlFJZ1JJdU1hcFVKOHVyVEdxblZGckFldy81aFdOZFdRUjNyN3FXQmlrYnk0aEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRpZ0F3SUJBZ0lJUnIzVlVFbTY4Yll3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU5UQXpOekFlRncweU16QXlNall5TXpRek5UZGFGdzB5TkRBeQpNall5TXpRek5UZGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUTE5SWdKZVJyUldOVGUKZC9UUWcwNWdNcWFMSU9oUU1HSUZKRytTSFJUTVl1RkxBUHhydGozV1g3T3hIRDFDaE5nM2p6VnJKbjcwYitpUApIRzJRNS9XTW8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVViNGJ3aEFWMitabXd0MWlaN2pKMGVDTytGRWd3Q2dZSUtvWkl6ajBFQXdJRFJ3QXcKUkFJZ2ZjNHJWdWJwUWVRMmxiVUJPNkxhK0hKNWVHbUFqSTRHMTV6K1pEZmZqQ1lDSUZGVkFTd1ZzVFRsUXlvMQpicmpsdnJzVW9rRmNxN3RIYjJxZk51dTV2aTB1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxXTnMKYVdWdWRDMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTFqYkdsbGJuUXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLTzZGUG4xd3dqbWFYdzc1T1l1blI1QytMbkpiRlpNYlgyWDRVR0kKMWFKK0tXb0tMZExmRnlZbVdvbHoyV3IzUnBhNGVxRXYySTg1bndINWp1NDZnNXlqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnZodkNFQlhiNW1iQzNXSm51Ck1uUjRJNzRVU0RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCeWR6VWRaMm5IZU56UHY2RkwzYUdabXMrZ2VuTmoKLzNaditGR1dhZi9nK3dJaEFNMnVCVE9maW9vQUptRXBpSi92WUFrQmRMdkdyWEdOY2R6UzdJRzRJR3BXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUQ4MXVnTGVmd0EvekE2U2l0V3czeDNiQWRHQ3A2eXJqYmNSRE14eFgwd3FvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTmZTSUNYa2EwVmpVM25mMDBJTk9ZREttaXlEb1VEQmlCU1J2a2gwVXpHTGhTd0Q4YTdZOQoxbCt6c1J3OVFvVFlONDgxYXlaKzlHL29qeHh0a09mMWpBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) - - -**Result:** pass - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.featureGates.RotateKubeletServerCertificate}' is present OR '{.featureGates.RotateKubeletServerCertificate}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJIZXg2aVo0Q09SYmlTd1pyYmRHa2ZIaHNKb2RZaGRScDFjNklkMngKS05ITHZzZEdxeEc0V2xySVp6WTEwTDlLWDJCVk9Senp0RUZ4cDc3MTlFTnVnNDZqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU1pFWGkwbDNsS3dXS1U3S0x6CkgwZG5aTzVEOWpBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCK2NQUUhLdXBJc05zU3BLVEg2NHJibFpKYTIyZ0oKdlh4TmNxb0dQSEdhUlFJZ1JJdU1hcFVKOHVyVEdxblZGckFldy81aFdOZFdRUjNyN3FXQmlrYnk0aEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRpZ0F3SUJBZ0lJUnIzVlVFbTY4Yll3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU5UQXpOekFlRncweU16QXlNall5TXpRek5UZGFGdzB5TkRBeQpNall5TXpRek5UZGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUTE5SWdKZVJyUldOVGUKZC9UUWcwNWdNcWFMSU9oUU1HSUZKRytTSFJUTVl1RkxBUHhydGozV1g3T3hIRDFDaE5nM2p6VnJKbjcwYitpUApIRzJRNS9XTW8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVViNGJ3aEFWMitabXd0MWlaN2pKMGVDTytGRWd3Q2dZSUtvWkl6ajBFQXdJRFJ3QXcKUkFJZ2ZjNHJWdWJwUWVRMmxiVUJPNkxhK0hKNWVHbUFqSTRHMTV6K1pEZmZqQ1lDSUZGVkFTd1ZzVFRsUXlvMQpicmpsdnJzVW9rRmNxN3RIYjJxZk51dTV2aTB1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxXTnMKYVdWdWRDMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTFqYkdsbGJuUXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLTzZGUG4xd3dqbWFYdzc1T1l1blI1QytMbkpiRlpNYlgyWDRVR0kKMWFKK0tXb0tMZExmRnlZbVdvbHoyV3IzUnBhNGVxRXYySTg1bndINWp1NDZnNXlqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnZodkNFQlhiNW1iQzNXSm51Ck1uUjRJNzRVU0RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCeWR6VWRaMm5IZU56UHY2RkwzYUdabXMrZ2VuTmoKLzNaditGR1dhZi9nK3dJaEFNMnVCVE9maW9vQUptRXBpSi92WUFrQmRMdkdyWEdOY2R6UzdJRzRJR3BXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUQ4MXVnTGVmd0EvekE2U2l0V3czeDNiQWRHQ3A2eXJqYmNSRE14eFgwd3FvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTmZTSUNYa2EwVmpVM25mMDBJTk9ZREttaXlEb1VEQmlCU1J2a2gwVXpHTGhTd0Q4YTdZOQoxbCt6c1J3OVFvVFlONDgxYXlaKzlHL29qeHh0a09mMWpBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{range .tlsCipherSuites[:]}{}{','}{end}' is present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJIZXg2aVo0Q09SYmlTd1pyYmRHa2ZIaHNKb2RZaGRScDFjNklkMngKS05ITHZzZEdxeEc0V2xySVp6WTEwTDlLWDJCVk9Senp0RUZ4cDc3MTlFTnVnNDZqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU1pFWGkwbDNsS3dXS1U3S0x6CkgwZG5aTzVEOWpBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCK2NQUUhLdXBJc05zU3BLVEg2NHJibFpKYTIyZ0oKdlh4TmNxb0dQSEdhUlFJZ1JJdU1hcFVKOHVyVEdxblZGckFldy81aFdOZFdRUjNyN3FXQmlrYnk0aEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRpZ0F3SUJBZ0lJUnIzVlVFbTY4Yll3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU5UQXpOekFlRncweU16QXlNall5TXpRek5UZGFGdzB5TkRBeQpNall5TXpRek5UZGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUTE5SWdKZVJyUldOVGUKZC9UUWcwNWdNcWFMSU9oUU1HSUZKRytTSFJUTVl1RkxBUHhydGozV1g3T3hIRDFDaE5nM2p6VnJKbjcwYitpUApIRzJRNS9XTW8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVViNGJ3aEFWMitabXd0MWlaN2pKMGVDTytGRWd3Q2dZSUtvWkl6ajBFQXdJRFJ3QXcKUkFJZ2ZjNHJWdWJwUWVRMmxiVUJPNkxhK0hKNWVHbUFqSTRHMTV6K1pEZmZqQ1lDSUZGVkFTd1ZzVFRsUXlvMQpicmpsdnJzVW9rRmNxN3RIYjJxZk51dTV2aTB1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxXTnMKYVdWdWRDMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTFqYkdsbGJuUXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLTzZGUG4xd3dqbWFYdzc1T1l1blI1QytMbkpiRlpNYlgyWDRVR0kKMWFKK0tXb0tMZExmRnlZbVdvbHoyV3IzUnBhNGVxRXYySTg1bndINWp1NDZnNXlqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnZodkNFQlhiNW1iQzNXSm51Ck1uUjRJNzRVU0RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCeWR6VWRaMm5IZU56UHY2RkwzYUdabXMrZ2VuTmoKLzNaditGR1dhZi9nK3dJaEFNMnVCVE9maW9vQUptRXBpSi92WUFrQmRMdkdyWEdOY2R6UzdJRzRJR3BXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUQ4MXVnTGVmd0EvekE2U2l0V3czeDNiQWRHQ3A2eXJqYmNSRE14eFgwd3FvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTmZTSUNYa2EwVmpVM25mMDBJTk9ZREttaXlEb1VEQmlCU1J2a2gwVXpHTGhTd0Q4YTdZOQoxbCt6c1J3OVFvVFlONDgxYXlaKzlHL29qeHh0a09mMWpBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -**Audit Script:** `check_for_default_sa.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` - -**Audit Execution:** - -```bash -./check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): serviceaccounts is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "serviceaccounts" in API group "" at the cluster scope Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "calico-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-fleet-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-impersonation-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cis-operator-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-node-lease" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-public" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "local" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "tigera-operator" true -``` - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -**Audit:** - -```bash -kubectl get psp global-restricted-psp -o json | jq -r '.spec.runAsUser.rule' -``` - -**Expected Result**: - -```console -'MustRunAsNonRoot' is present -``` - -**Returned Value**: - -```console -MustRunAsNonRoot -``` - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=5 -``` - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=5 -``` - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=2 -``` - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=4 -``` - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** pass - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=4 -``` - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -**Audit:** - -```bash -kubectl get psp global-restricted-psp -o json | jq -r .spec.requiredDropCapabilities[] -``` - -**Expected Result**: - -```console -'ALL' is present -``` - -**Returned Value**: - -```console -ALL -``` - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports Network Policies (Automated) - - -**Result:** pass - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -**Audit:** - -```bash -kubectl get pods --all-namespaces --selector='k8s-app in (calico-node, canal, cilium)' -o name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -**Audit Script:** `check_for_rke2_network_policies.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in kube-system kube-public default; do - policy_count=$(/var/lib/rancher/rke2/bin/kubectl get networkpolicy -n ${namespace} -o json | jq -r '.items | length') - if [ ${policy_count} -eq 0 ]; then - echo "false" - exit - fi -done - -echo "true" - -``` - -**Audit Execution:** - -```bash -./check_for_rke2_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is present -``` - -**Returned Value**: - -```console -true -``` - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md deleted file mode 100644 index 4d524d3fab4..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md +++ /dev/null @@ -1,3196 +0,0 @@ ---- -title: RKE2 Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.25 ---- - -This document is a companion to the [RKE2 Hardening Guide](../../../../pages-for-subheaders/rke2-hardening-guide.md), which provides prescriptive guidance on how to harden RKE2 clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.25 | - -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE2 install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -RKE2 launches control plane components as static pods, managed by the kubelet, and uses containerd as the container runtime. Configuration is defined by arguments passed to the container at the time of initialization or via configuration file. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE2 nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Master Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the -control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %a /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -644 -``` - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' -``` - -**Expected Result**: - -```console -'permissions' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 permissions=644 -``` - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root -``` - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/db/etcd -``` - -**Expected Result**: - -```console -permissions has permissions 700, expected 700 or more restrictive -``` - -**Returned Value**: - -```console -permissions=700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/db/etcd -``` - -**Expected Result**: - -```console -'etcd:etcd' is present -``` - -**Returned Value**: - -```console -etcd:etcd -``` - -### 1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 600 /etc/kubernetes/admin.conf - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/admin.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/admin.conf - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/admin.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 scheduler - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root scheduler - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 controllermanager - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/controller.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /var/lib/rancher/rke2/server/cred/controller.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/controller.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown -R root:root /var/lib/rancher/rke2/server/tls/ - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/tls -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 644 /var/lib/rancher/rke2/server/tls/*.crt - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.crt -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /var/lib/rancher/rke2/server/tls/*.key - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.key -``` - -**Expected Result**: - -```console -'permissions' is equal to '600' -``` - -**Returned Value**: - -```console -permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'AlwaysPullImages' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true root 15378 3910 99 23:32 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/readyz -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -### 1.2.33 Ensure that encryption providers are appropriately configured (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit:** - -```bash -/bin/sh -c 'if grep aescbc /var/lib/rancher/rke2/server/cred/encryption-config.json; then echo 0; fi' -``` - -**Expected Result**: - -```console -'0' is present -``` - -**Returned Value**: - -```console -{"kind":"EncryptionConfiguration","apiVersion":"apiserver.config.k8s.io/v1","resources":[{"resources":["secrets"],"providers":[{"aescbc":{"keys":[{"name":"aescbckey","secret":"TSpBkJhIU0sRx+84IZuBZ1qO+eaRdW31C7QCnF3+n8s="}]}},{"identity":{}}]}]} 0 -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4126 4014 0 23:27 ? 00:00:02 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 4126 4014 0 23:27 ? 00:00:02 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-25-112 ETCD_UNSUPPORTED_ARCH= POD_HASH=ab0b8a2ee7711940d3d951edece075f3 FILE_HASH=068666c5f959fc1023cb1761daaaed212727c04120747d825c78fd7683122e6d NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 HOME=/ -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-25-112 ETCD_UNSUPPORTED_ARCH= POD_HASH=ab0b8a2ee7711940d3d951edece075f3 FILE_HASH=068666c5f959fc1023cb1761daaaed212727c04120747d825c78fd7683122e6d NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 HOME=/ -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Audit Config:** - -```bash -cat /var/lib/rancher/rke2/server/db/etcd/config -``` - -**Expected Result**: - -```console -'ETCD_TRUSTED_CA_FILE' is present OR '{.peer-transport-security.trusted-ca-file}' is equal to '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-25-112 ETCD_UNSUPPORTED_ARCH= POD_HASH=ab0b8a2ee7711940d3d951edece075f3 FILE_HASH=068666c5f959fc1023cb1761daaaed212727c04120747d825c78fd7683122e6d NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 HOME=/ -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep | grep -o audit-policy-file -``` - -**Expected Result**: - -```console -'audit-policy-file' is equal to 'audit-policy-file' -``` - -**Returned Value**: - -```console -audit-policy-file -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive OR '/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is present OR '/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/kubelet.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit Script:** `check_cafile_permissions.sh` - -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -CAFILE=/node$CAFILE -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi - -``` - -**Audit Execution:** - -```bash -./check_cafile_permissions.sh -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit Script:** `check_cafile_ownership.sh` - -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -CAFILE=/node$CAFILE -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi - -``` - -**Audit Execution:** - -```bash -./check_cafile_ownership.sh -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /etc/rancher/rke2/rke2.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/rancher/rke2/rke2.yaml; then stat -c permissions=%a /etc/rancher/rke2/rke2.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /etc/rancher/rke2/rke2.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/rancher/rke2/rke2.yaml; then stat -c %U:%G /etc/rancher/rke2/rke2.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.makeIPTablesUtilChains}' is present OR '{.makeIPTablesUtilChains}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.eventRecordQPS}' is present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) - - -**Result:** pass - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.featureGates.RotateKubeletServerCertificate}' is present OR '{.featureGates.RotateKubeletServerCertificate}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{range .tlsCipherSuites[:]}{}{','}{end}' is present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -**Audit Script:** `check_for_default_sa.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` - -**Audit Execution:** - -```bash -./check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): serviceaccounts is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "serviceaccounts" in API group "" at the cluster scope Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "calico-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-fleet-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-impersonation-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cis-operator-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-node-lease" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-public" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "local" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "tigera-operator" true -``` - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -**Audit:** - -```bash -kubectl get psp global-restricted-psp -o json | jq -r '.spec.runAsUser.rule' -``` - -**Expected Result**: - -```console -'MustRunAsNonRoot' is present -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" -``` - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** fail - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -**Audit:** - -```bash -kubectl get psp global-restricted-psp -o json | jq -r .spec.requiredDropCapabilities[] -``` - -**Expected Result**: - -```console -'ALL' is present -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" -``` - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports Network Policies (Automated) - - -**Result:** pass - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -**Audit:** - -```bash -kubectl get pods --all-namespaces --selector='k8s-app in (calico-node, canal, cilium)' -o name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - - -**Result:** true - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -**Audit Script:** `check_for_rke2_network_policies.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in kube-system kube-public default; do - policy_count=$(/var/lib/rancher/rke2/bin/kubectl get networkpolicy -n ${namespace} -o json | jq -r '.items | length') - if [ ${policy_count} -eq 0 ]; then - echo "false" - exit - fi -done - -echo "true" - -``` - -**Audit Execution:** - -```bash -./check_for_rke2_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is present -``` - -**Returned Value**: - -```console -true -``` - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md new file mode 100644 index 00000000000..f14207e929d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md @@ -0,0 +1,3202 @@ +--- +title: RKE2 自我评估指南 - CIS Benchmark v1.24 - K8s v1.24 +--- + + + + + +本文档是 [RKE2 加固指南](rke2-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE2 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | + +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE2 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.24 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +RKE2 将 control plane 组件作为静态 Pod 启动，由 kubelet 管理，并使用 containerd 作为容器运行时。配置是由初始化时或通过配置文件传递给容器的参数定义的。 + +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE2 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 644 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %a /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' +``` + +**Expected Result**: + +```console +'600' is present +``` + +**Returned Value**: + +```console +644 +``` + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' +``` + +**Expected Result**: + +```console +'permissions' is equal to '600' +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' +``` + +**Expected Result**: + +```console +'644' is equal to '644' +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=644 +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root root:root +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/db/etcd +``` + +**Expected Result**: + +```console +permissions has permissions 700, expected 700 or more restrictive +``` + +**Returned Value**: + +```console +permissions=700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/db/etcd +``` + +**Expected Result**: + +```console +'etcd:etcd' is present +``` + +**Returned Value**: + +```console +etcd:etcd +``` + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/admin.conf + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/cred/admin.kubeconfig +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/admin.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 scheduler + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root scheduler + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 controllermanager + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/cred/controller.kubeconfig +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/server/cred/controller.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/controller.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /var/lib/rancher/rke2/server/tls/ + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/tls +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Automated) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 644 /var/lib/rancher/rke2/server/tls/*.crt + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.crt +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /var/lib/rancher/rke2/server/tls/*.key + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.key +``` + +**Expected Result**: + +```console +'permissions' is equal to '600' +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the --kubelet-https parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-https' is present OR '--kubelet-https' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.18 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxsize' is greater or equal to 100 +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true root 1017917 2489 83 16:16 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/readyz root 1017945 2489 75 16:16 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/livez +``` + +### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +### 1.2.33 Ensure that encryption providers are appropriately configured (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit:** + +```bash +/bin/sh -c 'if grep aescbc /var/lib/rancher/rke2/server/cred/encryption-config.json; then echo 0; fi' +``` + +**Expected Result**: + +```console +'0' is present +``` + +**Returned Value**: + +```console +{"kind":"EncryptionConfiguration","apiVersion":"apiserver.config.k8s.io/v1","resources":[{"resources":["secrets"],"providers":[{"aescbc":{"keys":[{"name":"aescbckey","secret":"akmMjUAq94YvsUftJpiA3b+9SClu0ESPBeckAI7KZBY="}]}},{"identity":{}}]}]} 0 +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2707 2593 0 Sep11 ? 00:06:20 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 2707 2593 0 Sep11 ? 00:06:20 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-10-113 ETCD_UNSUPPORTED_ARCH= NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=560b915d9afb672c20c2c1e8664bdf8f FILE_HASH=166aebdce42ff62fcdd2cefc9e7ed8f7c5b562d219ca6afec8f73adc654f65e7 HOME=/ +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-10-113 ETCD_UNSUPPORTED_ARCH= NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=560b915d9afb672c20c2c1e8664bdf8f FILE_HASH=166aebdce42ff62fcdd2cefc9e7ed8f7c5b562d219ca6afec8f73adc654f65e7 HOME=/ +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Audit Config:** + +```bash +cat /var/lib/rancher/rke2/server/db/etcd/config +``` + +**Expected Result**: + +```console +'ETCD_TRUSTED_CA_FILE' is present OR '{.peer-transport-security.trusted-ca-file}' is equal to '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-10-113 ETCD_UNSUPPORTED_ARCH= NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=560b915d9afb672c20c2c1e8664bdf8f FILE_HASH=166aebdce42ff62fcdd2cefc9e7ed8f7c5b562d219ca6afec8f73adc654f65e7 HOME=/ +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep | grep -o audit-policy-file +``` + +**Expected Result**: + +```console +'audit-policy-file' is equal to 'audit-policy-file' +``` + +**Returned Value**: + +```console +audit-policy-file +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'600' is present +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit Script:** `check_cafile_permissions.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +CAFILE=/node$CAFILE +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_permissions.sh +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit Script:** `check_cafile_ownership.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +CAFILE=/node$CAFILE +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_ownership.sh +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--protect-kernel-defaults' is equal to 'true' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.makeIPTablesUtilChains}' is present OR '{.makeIPTablesUtilChains}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +### 4.2.9 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Automated) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.eventRecordQPS}' is present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** pass + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.featureGates.RotateKubeletServerCertificate}' is present OR '{.featureGates.RotateKubeletServerCertificate}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{range .tlsCipherSuites[:]}{}{','}{end}' is present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Automated) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[]?.kind=="ServiceAccount" and .subjects[]?.name=="default") or (.subjects[]?.kind=="Group" and .subjects[]?.name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[]? != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +**Audit:** + +```bash +kubectl get psp global-restricted-psp && kubectl get psp global-restricted-psp -o json | jq -r ".spec.runAsUser.rule" || kubectl get psp restricted-noroot-psp && kubectl get psp restricted-noroot-psp -o json | jq -r ".spec.runAsUser.rule" +``` + +**Expected Result**: + +```console +'MustRunAsNonRoot' is equal to 'MustRunAsNonRoot' +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ Error from server (NotFound): podsecuritypolicies.policy "global-restricted-psp" not found Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES restricted-noroot-psp false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,csi,persistentVolumeClaim Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ MustRunAsNonRoot +``` + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=5 +``` + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=5 +``` + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=2 +``` + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=4 +``` + +### 5.2.7 Minimize the admission of root containers (Automated) + + +**Result:** pass + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=4 +``` + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +**Audit:** + +```bash +kubectl get psp global-restricted-psp && kubectl get psp global-restricted-psp -o json | jq -r .spec.requiredDropCapabilities[] || kubectl get psp restricted-noroot-psp && kubectl get psp restricted-noroot-psp -o json | jq -r .spec.requiredDropCapabilities[] +``` + +**Expected Result**: + +```console +'ALL' is equal to 'ALL' +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ Error from server (NotFound): podsecuritypolicies.policy "global-restricted-psp" not found Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES restricted-noroot-psp false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,csi,persistentVolumeClaim Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ ALL +``` + +### 5.2.9 Minimize the admission of containers with added capabilities (Automated) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports Network Policies (Automated) + + +**Result:** pass + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +**Audit:** + +```bash +kubectl get pods --all-namespaces --selector='k8s-app in (calico-node, canal, cilium)' -o name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +--count=1 +``` + +### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +**Audit Script:** `check_for_rke2_network_policies.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +for namespace in kube-system kube-public default; do + policy_count=$(/var/lib/rancher/rke2/bin/kubectl get networkpolicy -n ${namespace} -o json | jq -r '.items | length') + if [ ${policy_count} -eq 0 ]; then + echo "false" + exit + fi +done + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_rke2_network_policies.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md new file mode 100644 index 00000000000..a51c617eea8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md @@ -0,0 +1,2967 @@ +--- +title: RKE2 自我评估指南 - CIS Benchmark v1.7 - K8s v1.25/v1.26/v1.27 +--- + + + + + +本文档是 [RKE2 加固指南](rke2-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE2 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25/v1.26/v1.27 | + +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE2 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.7 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +RKE2 将 control plane 组件作为静态 Pod 启动，由 kubelet 管理，并使用 containerd 作为容器运行时。配置是由初始化时或通过配置文件传递给容器的参数定义的。 + +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE2 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then find /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml -name '*etcd*' | xargs stat -c permissions=%a; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=644 +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root root:root +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/db/etcd +``` + +**Expected Result**: + +```console +permissions has permissions 700, expected 700 or more restrictive +``` + +**Returned Value**: + +```console +permissions=700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** Not Applicable + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/admin.conf + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/cred/admin.kubeconfig +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/admin.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'600' is present +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /var/lib/rancher/rke2/server/cred/controller.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/server/cred/controller.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'600' is present +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/server/cred/controller.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/controller.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/tls +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /var/lib/rancher/rke2/server/tls/*.crt + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.crt +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /var/lib/rancher/rke2/server/tls/*.key + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.key +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.7 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.9 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.12 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.13 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true root 1018768 2419 99 16:17 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/readyz +``` + +### 1.2.14 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.15 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.16 Ensure that the --secure-port argument is not set to 0 - NoteThis recommendation is obsolete and will be deleted per the consensus process (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.17 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.18 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.19 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.20 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.21 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +### 1.2.22 Ensure that the --request-timeout argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, --request-timeout=300s + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--request-timeout' is not present OR '--request-timeout' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.23 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.24 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.25 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.26 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.27 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.28 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.29 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.30 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit:** + +```bash +ENCRYPTION_PROVIDER_CONFIG=$(ps -ef | grep kube-apiserver | grep -- --encryption-provider-config | sed 's%.*encryption-provider-config[= ]\([^ ]*\).*%\1%') if test -e $ENCRYPTION_PROVIDER_CONFIG; then grep -A1 'providers:' $ENCRYPTION_PROVIDER_CONFIG | tail -n1 | grep -o "[A-Za-z]*" | sed 's/^/provider=/'; fi +``` + +**Expected Result**: + +```console +'provider' is present +``` + +### 1.2.31 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 +Not Applicable. + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2645 2538 0 Sep11 ? 00:05:26 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 2645 2538 0 Sep11 ? 00:05:26 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= +Not Applicable. + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" +Not Applicable. + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-14-226 ETCD_UNSUPPORTED_ARCH= FILE_HASH=e9ca6f328e70dd3c17ba78c302ee32927a4961198e95d1d948114a5d7e350d99 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=aa1658a59ab75324ef67c786d08307e3 HOME=/ +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= +Not Applicable. + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true +Not Applicable. + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-14-226 ETCD_UNSUPPORTED_ARCH= FILE_HASH=e9ca6f328e70dd3c17ba78c302ee32927a4961198e95d1d948114a5d7e350d99 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=aa1658a59ab75324ef67c786d08307e3 HOME=/ +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Audit Config:** + +```bash +cat /var/lib/rancher/rke2/server/db/etcd/config +``` + +**Expected Result**: + +```console +'ETCD_TRUSTED_CA_FILE' is present OR '{.peer-transport-security.trusted-ca-file}' is equal to '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-14-226 ETCD_UNSUPPORTED_ARCH= FILE_HASH=e9ca6f328e70dd3c17ba78c302ee32927a4961198e95d1d948114a5d7e350d99 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=aa1658a59ab75324ef67c786d08307e3 HOME=/ +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +### 3.1.2 Service account token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of service account tokens. + +### 3.1.3 Bootstrap token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of bootstrap tokens. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not applicable. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit Script:** `check_cafile_permissions.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +CAFILE=/node$CAFILE +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_permissions.sh +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit Script:** `check_cafile_ownership.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +CAFILE=/node$CAFILE +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_ownership.sh +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.makeIPTablesUtilChains}' is present OR '{.makeIPTablesUtilChains}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.7 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable. + +### 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.eventRecordQPS}' is present OR '{.eventRecordQPS}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.10 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** pass + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.featureGates.RotateKubeletServerCertificate}' is present OR '{.featureGates.RotateKubeletServerCertificate}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{range .tlsCipherSuites[:]}{}{','}{end}' is present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.13 Ensure that a limit is set on pod PIDs (Manual) + + +**Result:** warn + +**Remediation:** +Decide on an appropriate level for this parameter and set it, +either via the --pod-max-pids command line parameter or the PodPidsLimit configuration file setting. + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.podPidsLimit}' is present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** warn + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +### 5.1.9 Minimize access to create persistent volumes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to PersistentVolume objects in the cluster. + +### 5.1.10 Minimize access to the proxy sub-resource of nodes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the proxy sub-resource of node objects. + +### 5.1.11 Minimize access to the approval sub-resource of certificatesigningrequests objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the approval sub-resource of certificatesigningrequest objects. + +### 5.1.12 Minimize access to webhook configuration objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the validatingwebhookconfigurations or mutatingwebhookconfigurations objects + +### 5.1.13 Minimize access to the service account token creation (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the token sub-resource of serviceaccount objects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +### 5.2.7 Minimize the admission of root containers (Manual) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Automated) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md deleted file mode 100644 index 252accbb79d..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md +++ /dev/null @@ -1,3196 +0,0 @@ ---- -title: RKE2 Self-Assessment Guide - CIS Benchmark v1.7 - K8s v1.25 ---- - -This document is a companion to the [RKE2 Hardening Guide](../../../../pages-for-subheaders/rke2-hardening-guide.md), which provides prescriptive guidance on how to harden RKE2 clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 | - -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE2 install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.7. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -RKE2 launches control plane components as static pods, managed by the kubelet, and uses containerd as the container runtime. Configuration is defined by arguments passed to the container at the time of initialization or via configuration file. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE2 nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Master Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the -control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %a /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -644 -``` - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' -``` - -**Expected Result**: - -```console -'permissions' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 permissions=644 -``` - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root -``` - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/db/etcd -``` - -**Expected Result**: - -```console -permissions has permissions 700, expected 700 or more restrictive -``` - -**Returned Value**: - -```console -permissions=700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/db/etcd -``` - -**Expected Result**: - -```console -'etcd:etcd' is present -``` - -**Returned Value**: - -```console -etcd:etcd -``` - -### 1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 600 /etc/kubernetes/admin.conf - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/admin.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/admin.conf - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/admin.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 scheduler - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root scheduler - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 controllermanager - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/controller.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /var/lib/rancher/rke2/server/cred/controller.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/controller.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown -R root:root /var/lib/rancher/rke2/server/tls/ - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/tls -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 644 /var/lib/rancher/rke2/server/tls/*.crt - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.crt -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /var/lib/rancher/rke2/server/tls/*.key - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.key -``` - -**Expected Result**: - -```console -'permissions' is equal to '600' -``` - -**Returned Value**: - -```console -permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'AlwaysPullImages' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true root 15378 3910 99 23:32 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/readyz -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -### 1.2.33 Ensure that encryption providers are appropriately configured (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit:** - -```bash -/bin/sh -c 'if grep aescbc /var/lib/rancher/rke2/server/cred/encryption-config.json; then echo 0; fi' -``` - -**Expected Result**: - -```console -'0' is present -``` - -**Returned Value**: - -```console -{"kind":"EncryptionConfiguration","apiVersion":"apiserver.config.k8s.io/v1","resources":[{"resources":["secrets"],"providers":[{"aescbc":{"keys":[{"name":"aescbckey","secret":"TSpBkJhIU0sRx+84IZuBZ1qO+eaRdW31C7QCnF3+n8s="}]}},{"identity":{}}]}]} 0 -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4126 4014 0 23:27 ? 00:00:02 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 4126 4014 0 23:27 ? 00:00:02 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-25-112 ETCD_UNSUPPORTED_ARCH= POD_HASH=ab0b8a2ee7711940d3d951edece075f3 FILE_HASH=068666c5f959fc1023cb1761daaaed212727c04120747d825c78fd7683122e6d NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 HOME=/ -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-25-112 ETCD_UNSUPPORTED_ARCH= POD_HASH=ab0b8a2ee7711940d3d951edece075f3 FILE_HASH=068666c5f959fc1023cb1761daaaed212727c04120747d825c78fd7683122e6d NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 HOME=/ -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Audit Config:** - -```bash -cat /var/lib/rancher/rke2/server/db/etcd/config -``` - -**Expected Result**: - -```console -'ETCD_TRUSTED_CA_FILE' is present OR '{.peer-transport-security.trusted-ca-file}' is equal to '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-25-112 ETCD_UNSUPPORTED_ARCH= POD_HASH=ab0b8a2ee7711940d3d951edece075f3 FILE_HASH=068666c5f959fc1023cb1761daaaed212727c04120747d825c78fd7683122e6d NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 HOME=/ -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep | grep -o audit-policy-file -``` - -**Expected Result**: - -```console -'audit-policy-file' is equal to 'audit-policy-file' -``` - -**Returned Value**: - -```console -audit-policy-file -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive OR '/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is present OR '/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/kubelet.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit Script:** `check_cafile_permissions.sh` - -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -CAFILE=/node$CAFILE -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi - -``` - -**Audit Execution:** - -```bash -./check_cafile_permissions.sh -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit Script:** `check_cafile_ownership.sh` - -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -CAFILE=/node$CAFILE -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi - -``` - -**Audit Execution:** - -```bash -./check_cafile_ownership.sh -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /etc/rancher/rke2/rke2.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/rancher/rke2/rke2.yaml; then stat -c permissions=%a /etc/rancher/rke2/rke2.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /etc/rancher/rke2/rke2.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/rancher/rke2/rke2.yaml; then stat -c %U:%G /etc/rancher/rke2/rke2.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.makeIPTablesUtilChains}' is present OR '{.makeIPTablesUtilChains}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.eventRecordQPS}' is present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) - - -**Result:** pass - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.featureGates.RotateKubeletServerCertificate}' is present OR '{.featureGates.RotateKubeletServerCertificate}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{range .tlsCipherSuites[:]}{}{','}{end}' is present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -**Audit Script:** `check_for_default_sa.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` - -**Audit Execution:** - -```bash -./check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): serviceaccounts is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "serviceaccounts" in API group "" at the cluster scope Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "calico-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-fleet-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-impersonation-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cis-operator-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-node-lease" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-public" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "local" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "tigera-operator" true -``` - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -**Audit:** - -```bash -kubectl get psp global-restricted-psp -o json | jq -r '.spec.runAsUser.rule' -``` - -**Expected Result**: - -```console -'MustRunAsNonRoot' is present -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" -``` - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** fail - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -**Audit:** - -```bash -kubectl get psp global-restricted-psp -o json | jq -r .spec.requiredDropCapabilities[] -``` - -**Expected Result**: - -```console -'ALL' is present -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" -``` - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports Network Policies (Automated) - - -**Result:** pass - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -**Audit:** - -```bash -kubectl get pods --all-namespaces --selector='k8s-app in (calico-node, canal, cilium)' -o name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - - -**Result:** true - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -**Audit Script:** `check_for_rke2_network_policies.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in kube-system kube-public default; do - policy_count=$(/var/lib/rancher/rke2/bin/kubectl get networkpolicy -n ${namespace} -o json | jq -r '.items | length') - if [ ${policy_count} -eq 0 ]; then - echo "false" - exit - fi -done - -echo "true" - -``` - -**Audit Execution:** - -```bash -./check_for_rke2_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is present -``` - -**Returned Value**: - -```console -true -``` - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/rancher-security-best-practices.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/rancher-security-best-practices.md new file mode 100644 index 00000000000..0d98495ffec --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/rancher-security-best-practices.md @@ -0,0 +1,21 @@ +--- +title: Rancher 安全最佳实践 +--- + + + + + +### 限制对 /version 和 /rancherversion 的公共访问 + +上游（本地） Rancher 实例提供正在运行的 Rancher 版本和用于构建它的 Go 版本信息。这些信息可以通过 `/version` 路径访问，该路径用于诸如自动化版本升级或确认部署成功等任务。上游实例还提供了可通过 `/rancherversion` 路径访问的 Rancher 版本信息。 + +攻击者可能会滥用这些信息来识别正在运行的 Rancher 版本，并与潜在的漏洞相关联以进行利用。如果你的上游 Rancher 实例在网上是公开可访问的，请使用 7 层防火墙来阻止 `/version` 和 `/rancherversion` 路径。 + +更多关于保护服务器的详细信息，请参阅 [OWASP Web Application Security Testing - Enumerate Infrastructure and Application Admin Interfaces](https://owasp.org/www-project-web-security-testing-guide/stable/4-Web_Application_Security_Testing/02-Configuration_and_Deployment_Management_Testing/05-Enumerate_Infrastructure_and_Application_Admin_Interfaces.html)。 + +### 会话管理 + +某些环境可能需要额外的安全控制来管理会话。例如，你可能希望限制用户的并发活动会话或限制可以从哪些地理位置发起这些会话。Rancher 默认情况下不支持这些功能。 + +如果你需要此类功能，请将 7 层防火墙与[外部认证](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md#外部认证与本地认证)结合使用。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/rancher-security.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/rancher-security.md new file mode 100644 index 00000000000..935aaa2b780 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/rancher-security.md @@ -0,0 +1,95 @@ +--- +title: 安全 +--- + + + + + + + + + + + +
+

安全策略

+

Rancher Labs 会负责任地披露问题，并致力于在合理的时间内解决所有问题。

+		 +

报告流程

+

请将安全问题发送至 security-rancher@suse.com。

+		 +

公告

+

订阅 Rancher 公告论坛以获取版本更新。

+
+ +安全是 Rancher 全部功能的基础。Rancher 集成了全部主流认证工具和服务，并提供了企业级的 [RBAC 功能](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md)，让你的 Kubernetes 集群更加安全。 + +本文介绍了安全相关的文档以及资源，让你的 Rancher 安装和下游 Kubernetes 集群更加安全。 + +### NeuVector 与 Rancher 的集成 + +NeuVector 是一个开源的、以容器为中心的安全应用程序，现已集成到 Rancher 中。NeuVector 提供生产安全、DevOps 漏洞保护和容器防火墙等功能。请参阅 [Rancher 文档](../../integrations-in-rancher/neuvector/neuvector.md) 和 [NeuVector 文档](https://open-docs.neuvector.com/)了解更多信息。 + +### 在 Kubernetes 集群上运行 CIS 安全扫描 + +Rancher 使用 [kube-bench](https://github.com/aquasecurity/kube-bench) 来运行安全扫描，从而检查 Kubernetes 是否按照 [CIS](https://www.cisecurity.org/cis-benchmarks/)（Center for Internet Security，互联网安全中心）Kubernetes Benchmark 中定义的安全最佳实践进行部署。 + +CIS Kubernetes Benchmark 是一个参考文档，用于为 Kubernetes 建立安全配置基线。 + +CIS 是一个 501(c\)(3) 非营利组织，成立于 2000 年 10 月，其使命是识别、开发、验证、促进和维持网络防御的最佳实践方案，并建立和指导社区，以在网络空间中营造信任的环境。 + +CIS Benchmark 是目标系统安全配置的最佳实践。CIS Benchmark 是由安全专家、技术供应商、公开和私人社区成员，以及 CIS Benchmark 开发团队共同志愿开发的。 + +Benchmark 提供两种类型的建议，分别是自动（Automated）和手动（Manual）。我们只运行 Automated 相关的测试。 + +Rancher 在集群上运行 CIS 安全扫描时会生成一份报告，该报告会显示每个测试的结果，包括测试概要以及 `passed`、`skipped` 和 `failed` 的测试数量。报告还包括失败测试的修正步骤。 + +有关详细信息，请参阅[安全扫描](../../how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md)。 + +### SELinux RPM + +[安全增强型 Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) 是对 Linux 的安全增强。被政府机构使用之后，SELinux 已成为行业标准，并在 CentOS 7 和 8 上默认启用。 + +我们提供了 `rancher-selinux` 和 `rke2-selinux` 两个 RPM（Red Hat 软件包），让 Rancher 产品能够在 SELinux 主机上正常运行。有关详细信息，请参阅[此页面](selinux-rpm/selinux-rpm.md)。 + +### Rancher 加固指南 + +Rancher 加固指南基于 CIS Kubernetes Benchmark。 + +加固指南为加固 Rancher 的生产安装提供了说明性指导。有关安全管控的完整列表，请参阅 Rancher 的 [CIS Kubernetes Benchmark 自我评估](#cis-benchmark-和自我评估)指南。 + +> 加固指南描述了如何保护集群中的节点，建议在安装 Kubernetes 之前参考加固指南中的步骤。 + +每个加固指南版本都针对特定的 CIS Kubernetes Benchmark、Kubernetes 和 Rancher 版本。 + +### CIS Benchmark 和自我评估 + +Benchmark 自我评估是 Rancher 安全加固指南的辅助。加固指南展示了如何加固集群，而 Benchmark 指南旨在帮助你评估加固集群的安全级别。 + +由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多管控验证检查都不适用。本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群的合规性。你可以前往 [CIS 网站](https://www.cisecurity.org/benchmark/kubernetes/)下载原始的 Benchmark 文档。 + +Rancher 自我评估指南的每个版本都对应于强化指南、Rancher、Kubernetes 和 CIS Benchmark 的特定版本。 + +### 第三方渗透测试报告 + +Rancher 会定期聘请第三方对 Rancher 软件栈进行安全审计和渗透测试。被测环境遵循 Rancher 在测试时提供的强化指南。以前的渗透测试报告如下。 + +结果： + +- [Cure53 渗透测试 - 2019 年 7 月](https://releases.rancher.com/documents/security/pen-tests/2019/RAN-01-cure53-report.final.pdf) +- [Untamed Theory 渗透测试 - 2019 年 3 月](https://releases.rancher.com/documents/security/pen-tests/2019/UntamedTheory-Rancher_SecurityAssessment-20190712_v5.pdf) + +请注意，新报告不再共享或公开发布。 + +### Rancher 安全公告和 CVE + +Rancher 致力于向社区通报我们产品中的安全问题。有关我们已解决的问题的 CVE（常见漏洞和暴露）列表，请参阅[此页](security-advisories-and-cves.md)。 + +### Kubernetes 安全最佳实践 + +有关保护 Kubernetes 集群的建议，请参阅 [Kubernetes 安全最佳实践](kubernetes-security-best-practices.md)指南。 + +### Rancher 安全最佳实践 + +有关保护 Rancher Manager 部署的建议，请参阅 [Rancher 安全最佳实践](rancher-security-best-practices.md)指南。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/rancher-webhook-hardening.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/rancher-webhook-hardening.md new file mode 100644 index 00000000000..e6c80bac05f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/rancher-webhook-hardening.md @@ -0,0 +1,137 @@ +--- +title: 加固 Rancher Webhook +--- + + + + + +Rancher Webhook 是 Rancher 中的一个重要组件，它在强制执行 Rancher 及其工作负载的安全要求方面发挥着重要的作用。为了减少其攻击面，对它的访问应限制为唯一有效的调用方：Kubernetes API server。这可以通过单独使用网络策略和认证，或相互结合使用以加固 webhook 免受攻击。 + +## 使用网络策略阻止外部流量 + +webhook 只应接受来自 Kubernetes API server 的请求。但是默认情况下，webhook 可以接受来自任何源的流量。如果你使用的是支持网络策略的 CNI，则可以创建一个策略来阻止来源不是 API server 的流量。 + +Kubernetes 中内置的网络策略资源无法阻止或允许来自集群主机的流量，并且 `kube-apiserver` 进程在主机网络上运行。因此你必须使用当前集群正在使用的 CNI 的高级网络策略资源。以下是 Calico 和 Cilium 的示例。更多详细的信息，请参阅你使用的 CNI 的文档。 + +### Calico + +使用 `crd.projectcalico.org/v1` API 组中的网络策略资源。使用 `app == 'rancher-webhook'` selector 为 webhook 创建一个规则，并且将 control plane 主机的 CIDR 设置为入口源： + +```yaml +apiVersion: crd.projectcalico.org/v1 +kind: NetworkPolicy +metadata: + name: allow-k8s + namespace: cattle-system +spec: + selector: app == 'rancher-webhook' + types: + - Ingress + ingress: + - action: Allow + protocol: TCP + source: + nets: + - 192.168.42.0/24 # control plane 主机的 CIDR。如果主机位于不同的子网中，则可能会列出多个。 + destination: + selector: + app == 'rancher-webhook' +``` + +### Cilium + +使用 `cilium.io/v2` API 组中的 CiliumNetworkPolicy 资源。将 `host` 和 `remote-node` 键添加到 `fromEntities` 入口规则。这会阻止集群内和外部流量，同时允许来自主机的流量。 + +```yaml +apiVersion: "cilium.io/v2" +kind: CiliumNetworkPolicy +metadata: + name: allow-k8s + namespace: cattle-system +spec: + endpointSelector: + matchLabels: + app: rancher-webhook + ingress: + - fromEntities: + - host + - remote-node +``` + +## 要求 Kubernetes API Server 对 Webhook 进行认证 + +Webhook 应仅接受来自 Kubernetes API server 的请求。默认情况下，webhook 不要求客户端对其进行认证。它可以接受任何请求。你可以配置 webhook 验证凭据，以便只有 API server 可以访问它。更多信息可以在 [Kubernetes 文档](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers)中找到。 + +1. 配置 API server 向 webhook 提供证书，指向 AdmissionConfiguration 文件以配置 ValidatingAdmissionWebhook 和 MutatingAdmissionWebhook 插件： + + ```yaml + # /etc/rancher/admission/admission.yaml + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: ValidatingAdmissionWebhook + configuration: + apiVersion: apiserver.config.k8s.io/v1 + kind: WebhookAdmissionConfiguration + kubeConfigFile: "/etc/rancher/admission/kubeconfig" + - name: MutatingAdmissionWebhook + configuration: + apiVersion: apiserver.config.k8s.io/v1 + kind: WebhookAdmissionConfiguration + kubeConfigFile: "/etc/rancher/admission/kubeconfig" + ``` + + 这也是配置其他准入插件的同一配置文件，例如 PodSecurity。如果你的发行版或设置使用了额外的准入插件，请将它们也配置在这里。例如，将 [RKE2 的 PodSecurity 配置](https://docs.rke2.io/security/pod_security_standards)添加到这个文件中。 + +2. 创建准入插件引用的 kubeconfig 文件。Rancher Webhook 仅支持客户端证书认证，因此请生成一个 TLS 密钥对，并且将 kubeconfig 设置为使用 `client-certificate` 和 `client-key` 或者使用 `client-certificate-data` 和 `client-key-data`。 例如： + + ```yaml + # /etc/rancher/admission/kubeconfig + apiVersion: v1 + kind: Config + users: + - name: 'rancher-webhook.cattle-system.svc' + user: + client-certificate: /path/to/client/cert + client-key: /path/to/client/key + ``` + +3. 使用 `--admission-control-config-file` 参数启动 kube-apiserver 服务，并将该参数值指向你的 AdmissionConfiguration 文件。具体的操作方式因发行版而异，并不受普遍支持，例如在托管的 Kubernetes 提供商中可能不支持。请查阅你的 Kubernetes 发行版的文档以获取详细的信息。 + + 对于 RKE2，可以使用这样的配置文件启动 `rke2-server`： + + ```yaml + # /etc/rancher/rke2/config.yaml + kube-apiserver-arg: + - admission-control-config-file=/etc/rancher/admission/admission.yaml + kube-apiserver-extra-mount: + - /etc/rancher/admission:/etc/rancher/admission:ro + ``` + + :::danger + 某些发行版会默认设置此参数。如果你的发行版预配置了它自己的 AdmissionConfiguration，则必须将其包含在你自定义的准入控制的配置文件中。例如，RKE2 在 `/etc/rancher/rke2/rke2-pss.yaml` 安装了一个 AdmissionConfiguration 文件，该文件配置了 PodSecurity 准入插件。在 config.yaml 中设置 `admission-control-config-file` 将会覆盖这个重要的安全设置。要包含两个插件，请参阅 [Default Pod Security Standards 文档](https://docs.rke2.io/security/pod_security_standards)并将相应的插件配置复制到你的 admission.yaml 中。 + ::: + +4. 如果你使用 Rancher 通过现有节点来配置你的集群，请在配置集群之前在节点上创建这些文件。 + + 如果你使用 Rancher 在新节点上配置你的集群，请先允许集群配置完成，然后使用 Rancher 提供的 SSH key 和 IP 地址连接到节点，并将 RKE2 的配置文件放在 `/etc/rancher/rke2/config.yaml.d/` 目录下。 + +5. 使用这些凭证配置集群后，配置 Rancher cluster agent 以在 webhook 中启用认证。创建一个包含以下内容的 chart values 的文件： + + ```yaml + # values.yaml + auth: + clientCA: + allowedCNs: + - + - + ``` + +6. 在配置的集群中的 `cattle-system` 命名空间下创建一个 configmap，并包含这些 values 配置： + + ``` + kubectl --namespace cattle-system create configmap rancher-config --from-file=rancher-webhook=values.yaml + ``` + + webhook 将会使用这些 values 配置重新启动。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/security-advisories-and-cves.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/security-advisories-and-cves.md index c2baac88dcf..12bdf210497 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/security-advisories-and-cves.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/security-advisories-and-cves.md @@ -2,10 +2,19 @@ title: 安全公告和 CVE --- + + + + Rancher 致力于向社区披露我们产品的安全问题。我们会针对已解决的问题发布安全公告和 CVE（Common Vulnerabilities and Exposures，通用漏洞披露）。Rancher GitHub 上的[安全页面](https://github.com/rancher/rancher/security/advisories)也会发布新的安全公告。 | ID | 描述 | 日期 | 解决 | |----|-------------|------|------------| +| [CVE-2024-22030](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-22030) | 发现了 Rancher 和 Fleet 代理的一个漏洞，目前被认为是中到高危的 CVE。在非特定情况下，这个漏洞允许恶意行为者接管现有的 Rancher 节点。攻击者需要控制一个过期的域名，或者对该域名执行 DNS 欺骗/劫持攻击才可以利用此漏洞。被攻击的域名是 Rancher URL（用作 Rancher 集群的 server-url）。目前还没有可用的修复方案，它影响所有受支持的 Rancher 版本。建议客户和用户遵循我们[博客文章](https://www.suse.com/c/rancher-security-update/)中描述的建议和最佳实践。 | 2024 年 2 月 16 日 | 处理中 | +| [CVE-2023-32193](https://github.com/rancher/norman/security/advisories/GHSA-r8f4-hv23-6qp6) | 在 Rancher 2.6.13、2.7.9 和 2.8.1 及之前的版本中发现了一个问题。多个 Cross-Site Scripting (XSS) 漏洞可通过 Rancher UI (Norman) 进行利用。 | 2024 年 2 月 8 日 | Rancher [v2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2)、[v2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) 和 [v2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | +| [CVE-2023-32192](https://github.com/rancher/apiserver/security/advisories/GHSA-833m-37f7-jq55) | 在 Rancher 2.6.13、2.7.9 和 2.8.1 及之前的版本中发现了一个问题。多个 Cross-Site Scripting (XSS) 漏洞，可以通过 Rancher UI (Apiserver) 进行利用 | 2024 年 2 月 8 日 | Rancher [v2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2)、[v2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) 和 [v2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | +| [CVE-2023-22649](https://github.com/rancher/rancher/security/advisories/GHSA-xfj7-qf8w-2gcr) | 在 Rancher 2.6.13、2.7.9 和 2.8.1 及之前的版本中发现了一个问题。敏感数据可能会泄漏到 Rancher 的审计日志中。 | 2024 年 2 月 8 日 | Rancher [v2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2)、[v2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) 和 [v2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | +| [CVE-2023-32194](https://github.com/rancher/rancher/security/advisories/GHSA-c85r-fwc7-45vc) | 在 Rancher 2.6.13、2.7.9 和 2.8.1 及之前的版本中发现了一个问题。当为 “namespace” 资源类型授予 `create` 或 `*` 全局角色时，任何 API 组中拥有权限的用户可以管理核心 API 组中的 namespace。 | 2024 年 2 月 8 日 | Rancher [v2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2)、[v2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) 和 [v2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | | [CVE-2023-22648](https://github.com/rancher/rancher/security/advisories/GHSA-vf6j-6739-78m8) | 在 Rancher 2.6.12 和 2.7.3 及之前的版本中发现了一个问题。在用户注销并重新登录到 Rancher UI 之前，Azure AD 中的权限更改不会反映给用户。 | 2023 年 5 月 31 日 | Rancher [v2.7.4](https://github.com/rancher/rancher/releases/tag/v2.7.4) | | [CVE-2022-43760](https://github.com/rancher/rancher/security/advisories/GHSA-46v3-ggjg-qq3x) | 在 Rancher 2.6.12 和 2.7.3 及之前的版本中发现了一个问题。攻击者可以通过 Rancher UI 利用多个跨站脚本 (XSS) 漏洞。 | 2023 年 5 月 31 日 | Rancher [v2.7.4](https://github.com/rancher/rancher/releases/tag/v2.7.4) | | [CVE-2020-10676](https://github.com/rancher/rancher/security/advisories/GHSA-8vhc-hwhc-cpj4) | 在 Rancher 2.6.12 和 2.7.3 及之前的版本中发现了一个问题。具有更新命名空间权限的用户可以将该命名空间移动到他们无权访问的项目中。 | 2023 年 5 月 31 日 | Rancher [v2.7.4](https://github.com/rancher/rancher/releases/tag/v2.7.4) | diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/selinux-rpm/selinux-rpm.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/selinux-rpm/selinux-rpm.md new file mode 100644 index 00000000000..2c96bf912ee --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-security/selinux-rpm/selinux-rpm.md @@ -0,0 +1,20 @@ +--- +title: SELinux RPM +--- + + + + + +[安全增强型 Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) 是对 Linux 的安全增强。 + +它由 Red Hat 开发，是 Linux 上 MAC（mandatory access controls，强制访问控制）的实现。系统管理员可以使用 MAC 设置应用程序和用户是如何访问不同资源的，例如文件、设备、网络和进程间的通信。SELinux 还通过默认限制操作系统来增强安全性。 + +被政府机构使用之后，SELinux 已成为行业标准，并在 CentOS 7 和 8 上默认启用。要检查 SELinux 是否在你的系统上启用和执行，请使用 `getenforce`： + +``` +# getenforce +Enforcing +``` + +我们提供了 [`rancher-selinux`](about-rancher-selinux.md) 和 [`rke2-selinux`](about-rke2-selinux.md) 两个 RPM（Red Hat 软件包），让 Rancher 产品能够在 SELinux 主机上正常运行。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/user-settings/user-settings.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/user-settings/user-settings.md new file mode 100644 index 00000000000..47ff9cb2518 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/user-settings/user-settings.md @@ -0,0 +1,19 @@ +--- +title: 用户设置 +--- + + + + + +在 Rancher 中，每个用户都有很多与登录相关的设置，例如个人偏好、API 密钥等。你可以从**用户设置**菜单中配置这些设置。你可以单击主菜单中的头像来打开此菜单。 + +![用户设置菜单](/img/user-settings.png) + +可用的用户设置包括： + +- [API & 密钥](api-keys.md)：如果你想以编程方式与 Rancher 交互，你需要一个 API 密钥。你可以按照本节中的说明获取密钥。 +- [云凭证](manage-cloud-credentials.md)：管理[节点模板](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md#节点模板)使用的云凭证，从而[为集群配置节点](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)。 +- [节点模板](manage-node-templates.md)：管理 [Rancher 用来为集群配置节点](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)的模板。 +- [偏好设置](user-preferences.md)：设置 Rancher UI 的表面首选项。 +- 登出：结束你的用户会话。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/about-the-api/about-the-api.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/about-the-api/about-the-api.md new file mode 100644 index 00000000000..f1756eb6e5d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/about-the-api/about-the-api.md @@ -0,0 +1,93 @@ +--- +title: API +--- + + + + + +## 如何使用 API + +API 有自己的用户界面，你可以从 Web 浏览器访问它。这是查看资源、执行操作以及查看等效 cURL 或 HTTP 请求和响应的一种简单的方法。要访问它： + + + + +1. 单击右上角的用户头像。 +1. 单击**账号 & API 密钥**。 +1. 在 **API 密钥**下，找到 **API 端点**字段并单击链接。该链接类似于 `https:///v3`，其中 `` 是 Rancher deployment 的完全限定域名。 + + + + +转到位于 `https:///v3` 的 URL 端点，其中 `` 是你的 Rancher deployment 的完全限定域名。 + + + + +## 认证 + +API 请求必须包含认证信息。认证是通过 [API 密钥](../user-settings/api-keys.md)使用 HTTP 基本认证完成的。API 密钥可以创建新集群并通过 `/v3/clusters/` 访问多个集群。[集群和项目角色](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md)会应用于这些键，并限制账号可以查看的集群和项目以及可以执行的操作。 + +默认情况下，某些集群级别的 API 令牌是使用无限期 TTL（`ttl=0`）生成的。换言之，除非你让令牌失效，否则 `ttl=0` 的 API 令牌永远不会过期。有关如何使 API 令牌失效的详细信息，请参阅 [API 令牌](api-tokens.md)。 + +## 发出请求 + +该 API 通常是 RESTful 的，但是还具有多种功能。这些功能可以使客户端发现所有内容，因此可以编写通用客户端，而不必为每种资源编写特定代码。有关通用 API 规范的详细信息，请参阅[此处](https://github.com/rancher/api-spec/blob/master/specification.md)。 + +- 每种类型都有一个 Schema，这个 Schema 描述了以下内容： + - 用于获取此类资源集合的 URL + - 资源可以具有的每个字段及其类型、基本验证规则、是必填还是可选字段等 + - 在此类资源上可以执行的每个操作，以及它们的输入和输出（也作为 schema） + - 允许过滤的每个字段 + - 集合本身或集合中的单个资源可以使用的 HTTP 操作方法 + + +- 因此，你可以只加载 schema 列表并了解 API 的所有信息。实际上，这是 API 的 UI 工作方式，它不包含特定于 Rancher 本身的代码。每个 HTTP 响应中的 `X-Api-Schemas` 标头都会发送获取 Schemas 的 URL。你可以按照每个 schema 上的 `collection` 链接了解要在哪里列出资源，并在返回资源中的其他 `links` 中获取其他信息。 + +- 在实践中，你可能只想构造 URL 字符串。我们强烈建议将此限制为在顶层列出的集合 (`/v3/`)，或获取特定资源 (`/v3//`)。除此之外的任何内容都可能在将来的版本中发生更改。 + +- 资源之间相互之间有联系，称为链接（links）。每个资源都包含一个 `links` 映射，其中包含链接名称和用于检索该信息的 URL。同样，你应该 `GET` 资源并遵循 `links` 映射中的 URL，而不是自己构造这些字符串。 + +- 大多数资源都有操作（action），表示可以执行某个操作或改变资源的状态。要使用操作，请将 HTTP `POST` 请求发送到 `actions` 映射中你想要的操作的 URL。某些操作需要输入或生成输出，请参阅每种类型的独立文档或 schema 以获取具体信息。 + +- 要编辑资源，请将 HTTP `PUT` 请求发送到资源上的 `links.update` 链接，其中包含要更改的字段。如果链接丢失，则你无权更新资源。未知字段和不可编辑的字段将被忽略。 + +- 要删除资源，请将 HTTP `DELETE` 请求发送到资源上的 `links.remove` 链接。如果链接丢失，则你无权更新资源。 + +- 要创建新资源，HTTP `POST` 到 schema（即 `/v3/`）中的集合 URL。 + +## 过滤 + +你可以使用 HTTP 查询参数的公共字段在服务器端过滤大多数集合。`filters` 映射显示了可以过滤的字段，以及过滤后的值在你发起的请求中是什么。API UI 具有设置过滤和显示适当请求的控件。对于简单的 "equals" 匹配，它只是 `field=value`。你可以将修饰符添加到字段名称，例如 `field_gt=42` 表示“字段大于 42”。详情请参阅 [API 规范](https://github.com/rancher/api-spec/blob/master/specification.md#filtering)。 + +## 排序 + +你可以使用 HTTP 查询参数的公共字段在服务器端排序大多数集合。`sortLinks` 映射显示了可用的排序，以及用于获取遵循该排序的集合的 URL。它还包括当前响排序依据的信息（如果指定）。 + +## 分页 + +默认情况下，API 响应以每页 100 个资源的限制进行分页。你可以通过 `limit` 查询参数进行更改，最大为 1000，例如 `/v3/pods?limit=1000`。集合响应中的 `pagination` 映射能让你知道你是否拥有完整的结果集，如果没有，则会指向下一页的链接。 + +## 捕获 Rancher API 调用 + +你可以使用浏览器开发人员工具来捕获 Rancher API 的调用方式。例如，你可以按照以下步骤使用 Chrome 开发人员工具来获取用于配置 RKE 集群的 API 调用： + +1. 在 Rancher UI 中，转到**集群管理**并单击**创建**。 +1. 单击某个集群类型。此示例使用 Digital Ocean。 +1. 使用集群名称和节点模板填写表单，但不要单击**创建**。 +1. 在创建集群之前，你需要打开开发人员工具才能看到正在记录的 API 调用。要打开工具，右键单击 Rancher UI，然后单击**检查**。 +1. 在开发者工具中，单击 **Network** 选项卡。 +1. 在 **Network** 选项卡上，确保选择了 **Fetch/XHR**。 +1. 在 Rancher UI 中，单击**创建**。在开发者工具中，你应该会看到一个名为 `cluster?_replace=true` 的新网络请求。 +1. 右键单击 `cluster?_replace=true` 并单击**复制 > 复制为 cURL**。 +1. 将结果粘贴到文本编辑器中。你将能够看到 POST 请求，包括被发送到的 URL、所有标头以及请求的完整正文。此命令可用于从命令行创建集群。请注意，请求包含凭证，因此请将请求存储在安全的地方。 + +### 启用在 API 中查看 + +你还可以查看针对各自集群和资源捕获的 Rancher API 调用。 默认情况下不启用此功能。 要启用它： + +1. 单击 UI 右上角的 **用户图标**，然后从下拉菜单中选择 **偏好设置** +1. 在**高级功能**部分下，单击**启用"在 API 中查看"** + +选中后，**在 API 中查看**链接现在将显示在 UI 资源页面上的 **⋮** 子菜单下。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cli-with-rancher/cli-with-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cli-with-rancher/cli-with-rancher.md new file mode 100644 index 00000000000..51f50ec2dca --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cli-with-rancher/cli-with-rancher.md @@ -0,0 +1,9 @@ +--- +title: Rancher CLI +--- + + + + + +Rancher CLI 是一个命令行工具，用于在工作站中与 Rancher 进行交互。以下文档将描述 [Rancher CLI](rancher-cli.md) 和 [kubectl实用程序](kubectl-utility.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/hardening-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/hardening-guides.md new file mode 100644 index 00000000000..6de7d34574f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/hardening-guides.md @@ -0,0 +1,55 @@ +--- +title: Rancher 自我评估和加固指南 +--- + + + + + +Rancher 为每个受支持的 Rancher 版本的 Kubernetes 发行版提供了特定的安全强化指南。 + +## Rancher Kubernetes 发行版 + +Rancher 使用以下 Kubernetes 发行版： + +- [**RKE**](https://rancher.com/docs/rke/latest/en/)（Rancher Kubernetes Engine）是经过 CNCF 认证的 Kubernetes 发行版，完全在 Docker 容器中运行。 +- [**RKE2**](https://docs.rke2.io/) 是一个完全合规的 Kubernetes 发行版，专注于安全和合规性。 +- [**K3s**](https://docs.k3s.io/) 是一个完全合规的，轻量级 Kubernetes 发行版。它易于安装，内存需求只有上游 Kubernetes 的一半，所有组件都在一个小于 100 MB 的二进制文件中。 + +要加固运行未列出的发行版的 Kubernetes 集群，请参阅 Kubernetes 提供商文档。 + +## 加固指南和 Benchmark 版本 + +每个自我评估指南都附有强化指南。这些指南与列出的 Rancher 版本一起进行了测试。每个自我评估指南都在特定的 Kubernetes 版本和 CIS Benchmark 版本上进行了测试。如果 CIS Benchmark 尚未针对你的 Kubernetes 版本进行验证，你可以使用现有指南，直到添加适合你的版本的指南。 + +### RKE 指南 + +| Kubernetes 版本 | CIS Benchmark 版本 | 自我评估指南 | 加固指南 | +|--------------------|-----------------------|-----------------------|------------------| +| Kubernetes v1.23 | CIS v1.23 | [链接](rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md) | [链接](rke1-hardening-guide/rke1-hardening-guide.md) | +| Kubernetes v1.24 | CIS v1.24 | [链接](rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md) | [链接](rke1-hardening-guide/rke1-hardening-guide.md) | +| Kubernetes v1.25/v1.26/v1.27 | CIS v1.7 | [链接](rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md) | [链接](rke1-hardening-guide/rke1-hardening-guide.md) | + +### RKE2 指南 + +| 类型 | Kubernetes 版本 | CIS Benchmark 版本 | 自我评估指南 | 加固指南 | +|------|--------------------|-----------------------|-----------------------|------------------| +| Rancher provisioned RKE2 | Kubernetes v1.23 | CIS v1.23 | [链接](rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md) | [链接](rke2-hardening-guide/rke2-hardening-guide.md) | +| Rancher provisioned RKE2 | Kubernetes v1.24 | CIS v1.24 | [链接](rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md) | [链接](rke2-hardening-guide/rke2-hardening-guide.md) | +| Rancher provisioned RKE2 | Kubernetes v1.25/v1.26/v1.27 | CIS v1.7 | [链接](rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md) | [链接](rke2-hardening-guide/rke2-hardening-guide.md) | +| Standalone RKE2 | Kubernetes v1.25/v1.26/v1.27 | CIS v1.7 | [链接](https://docs.rke2.io/security/cis_self_assessment123) | [链接](https://docs.rke2.io/security/hardening_guide) | + +### K3s 指南 + +| 类型 | Kubernetes 版本 | CIS Benchmark 版本 | 自我评估指南 | 加固指南 | +|------|--------------------|-----------------------|-----------------------|------------------| +| Rancher provisioned K3s cluster | Kubernetes v1.23 | CIS v1.23 | [链接](k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md) | [链接](k3s-hardening-guide/k3s-hardening-guide.md) | +| Rancher provisioned K3s cluster | Kubernetes v1.24 | CIS v1.24 | [链接](k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md) | [链接](k3s-hardening-guide/k3s-hardening-guide.md) | +| Rancher provisioned K3s cluster | Kubernetes v1.25/v1.26/v1.27 | CIS v1.7 | [链接](k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md) | [链接](k3s-hardening-guide/k3s-hardening-guide.md) | +| Standalone K3s | Kubernetes v1.22 up to v1.24 | CIS v1.23 | [链接](https://docs.k3s.io/security/self-assessment) | [链接](https://docs.k3s.io/security/hardening-guide) | + +## 在 SELinux 上使用 Rancher + +[Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) 是一个内核模块，为 Linux 添加了额外的访问控制和安全工具。SELinux 过去曾被政府机构使用，现在已成为行业标准。SELinux 在 RHEL 和 CentOS 上默认启用。 + +要将 Rancher 与 SELinux 结合使用，我们建议[安装](../selinux-rpm/about-rancher-selinux.md) `rancher-selinux`。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md new file mode 100644 index 00000000000..d40edd47e96 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md @@ -0,0 +1,744 @@ +--- +title: K3s 加固指南 +--- + + + + + +本文档提供了针对生产环境的 K3s 集群进行加固的具体指导，以便在使用 Rancher 部署之前进行配置。它概述了满足信息安全中心（Center for Information Security, CIS）Kubernetes benchmark controls 所需的配置和控制。 + +:::note +这份加固指南描述了如何确保你集群中的节点安全。我们建议你在安装 Kubernetes 之前遵循本指南。 +::: + +此加固指南适用于 K3s 集群，并与以下版本的 CIS Kubernetes Benchmark、Kubernetes 和 Rancher 相关联： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|------------------------------| +| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 至 v1.26 | + +:::note +在 Benchmark v1.7 中，不再需要 `--protect-kernel-defaults` (4.2.6) 参数，并已被 CIS 删除。 +::: + +有关如何评估加固的 K3s 集群与官方 CIS benchmark 的更多细节，请参考特定 Kubernetes 和 CIS benchmark 版本的 K3s 自我评估指南。 + +K3s 在不需要修改的情况下通过了许多 Kubernetes CIS controls，因为它默认应用了几个安全缓解措施。然而，有一些值得注意的例外情况，需要手动干预才能完全符合 CIS Benchmark 要求： + +1. K3s 不修改主机操作系统。任何主机级别的修改都需要手动完成。 +2. 某些 CIS policy controls，例如 `NetworkPolicies` 和 `PodSecurityStandards`（在 v1.24 及更早版本中为 `PodSecurityPolicies`），会限制集群功能。 + 你必须选择让 K3s 配置这些策略。在你的命令行标志或配置文件中添加相应的选项（启用准入插件），并手动应用适当的策略。 + 请参阅以下详细信息。 + +CIS Benchmark 的第一部分（1.1）主要关注于 Pod manifest 的权限和所有权。由于发行版中的所有内容都打包在一个二进制文件中，因此这一部分不适用于 K3s 的核心组件。 + +## 主机级别要求 + +### 确保 `protect-kernel-defaults`已经设置 + + + + +自 CIS benchmark 1.7 开始，不再需要`protect-kernel-defaults`。 + + + + +这是一个 kubelet 标志，如果所需的内核参数未设置或设置为与 kubelet 的默认值不同的值，将导致 kubelet 退出。 + +可以在 Rancher 的集群配置中设置 `protect-kernel-defaults` 标志。 + +```yaml +spec: + rkeConfig: + machineSelectorConfig: + - config: + protect-kernel-defaults: true +``` + + + + +### 设置内核参数 + +建议为集群中的所有节点类型设置以下 `sysctl` 配置。在 `/etc/sysctl.d/90-kubelet.conf` 中设置以下参数： + +```ini +vm.panic_on_oom=0 +vm.overcommit_memory=1 +kernel.panic=10 +kernel.panic_on_oops=1 +``` + +运行 `sudo sysctl -p /etc/sysctl.d/90-kubelet.conf` 以启用设置。 + +此配置需要在设置 kubelet 标志之前完成，否则 K3s 将无法启动。 + +## Kubernetes 运行时要求 + +CIS Benchmark 的运行时要求主要围绕 Pod 安全（通过 PSP 或 PSA）、网络策略和 API 服务器审计日志展开。 + +默认情况下，K3s 不包含任何 Pod 安全或网络策略。然而，K3s 附带一个控制器，可以强制执行你创建的任何网络策略。默认情况下，K3s 启用了 `PodSecurity` 和 `NodeRestriction` 等多个准入控制器。 + +### Pod 安全 + + + + +K3s v1.25 及更新版本支持 [Pod 安全准入(PSA)](https://kubernetes.io/docs/concepts/security/pod-security-admission/)，用于控制 Pod 安全性。 + +你可以在 Rancher 中通过集群配置，设置 `defaultPodSecurityAdmissionConfigurationTemplateName` 字段来指定 PSA 配置： + +```yaml +spec: + defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted +``` + +Rancher 提供了 `rancher-restricted` 模板，用于强制执行高度限制性的 Kubernetes 上游 [`Restricted`](https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted) 配置文件，其中包含了 Pod 加固的最佳实践。 + + + + +K3s v1.24 及更早版本支持 [Pod 安全策略 (PSP)](https://v1-24.docs.kubernetes.io/docs/concepts/security/pod-security-policy/) 以控制 Pod 安全性。 + +你可以在 Rancher 中通过集群配置，传递以下标志来启用 PSPs： + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount +``` + +这会保留 `NodeRestriction` 插件并启用 `PodSecurityPolicy`。 + +启用 PSPs 后，你可以应用策略来满足 CIS Benchmark 第 5.2 节中描述的必要控制。 + +:::note +这些是 CIS Benchmark 中的手动检查。CIS 扫描结果将标记为 `warning`，因为需要集群操作员进行手动检查。 +::: + +以下是合规的 PSP 示例： + +```yaml +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: restricted-psp +spec: + privileged: false # CIS - 5.2.1 + allowPrivilegeEscalation: false # CIS - 5.2.5 + requiredDropCapabilities: # CIS - 5.2.7/8/9 + - ALL + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'downwardAPI' + - 'csi' + - 'persistentVolumeClaim' + - 'ephemeral' + hostNetwork: false # CIS - 5.2.4 + hostIPC: false # CIS - 5.2.3 + hostPID: false # CIS - 5.2.2 + runAsUser: + rule: 'MustRunAsNonRoot' # CIS - 5.2.6 + seLinux: + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + - min: 1 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + - min: 1 + max: 65535 + readOnlyRootFilesystem: false +``` + +要使示例 PSP 生效，我们需要创建一个 `ClusterRole` 和 一个`ClusterRoleBinding`。我们还需要为需要额外权限的系统级 Pod 提供“系统无限制策略”，以及允许必要的 sysctls 来实现 ServiceLB 完整功能的额外策略。 + +```yaml +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: restricted-psp +spec: + privileged: false + allowPrivilegeEscalation: false + requiredDropCapabilities: + - ALL + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'downwardAPI' + - 'csi' + - 'persistentVolumeClaim' + - 'ephemeral' + hostNetwork: false + hostIPC: false + hostPID: false + runAsUser: + rule: 'MustRunAsNonRoot' + seLinux: + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + - min: 1 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + - min: 1 + max: 65535 + readOnlyRootFilesystem: false +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: system-unrestricted-psp + annotations: + seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*' +spec: + allowPrivilegeEscalation: true + allowedCapabilities: + - '*' + fsGroup: + rule: RunAsAny + hostIPC: true + hostNetwork: true + hostPID: true + hostPorts: + - max: 65535 + min: 0 + privileged: true + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + volumes: + - '*' +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: svclb-psp + annotations: + seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*' +spec: + allowPrivilegeEscalation: false + allowedCapabilities: + - NET_ADMIN + allowedUnsafeSysctls: + - net.ipv4.ip_forward + - net.ipv6.conf.all.forwarding + fsGroup: + rule: RunAsAny + hostPorts: + - max: 65535 + min: 0 + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:restricted-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + verbs: + - use + resourceNames: + - restricted-psp +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:system-unrestricted-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - system-unrestricted-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:svclb-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - svclb-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:svc-local-path-provisioner-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - system-unrestricted-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:svc-coredns-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - system-unrestricted-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:svc-cis-operator-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - system-unrestricted-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: default:restricted-psp +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted-psp +subjects: +- kind: Group + name: system:authenticated + apiGroup: rbac.authorization.k8s.io +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: system-unrestricted-node-psp-rolebinding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:system-unrestricted-psp +subjects: +- apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:nodes +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: system-unrestricted-svc-acct-psp-rolebinding + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:system-unrestricted-psp +subjects: +- apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: svclb-psp-rolebinding + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:svclb-psp +subjects: +- kind: ServiceAccount + name: svclb +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: svc-local-path-provisioner-psp-rolebinding + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:svc-local-path-provisioner-psp +subjects: +- kind: ServiceAccount + name: local-path-provisioner-service-account +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: svc-coredns-psp-rolebinding + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:svc-coredns-psp +subjects: +- kind: ServiceAccount + name: coredns +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: svc-cis-operator-psp-rolebinding + namespace: cis-operator-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:svc-cis-operator-psp +subjects: +- kind: ServiceAccount + name: cis-operator-serviceaccount +``` + +上述策略可以放置在 `/var/lib/rancher/k3s/server/manifests` 目录下名为 `policy.yaml` 的文件中。在启动 K3s 之前，必须创建策略文件和其目录结构。建议限制访问权限以避免泄露潜在的敏感信息。 + +```shell +sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/manifests +``` + +:::note +CNI、DNS 和 Ingress 等关键 Kubernetes 组件在 `kube-system` 命名空间中作为 Pod 运行。因此，这个命名空间的限制政策较少，从而使这些组件能够正常运行。 +::: + + + + +### 网络策略 + +CIS 要求所有命名空间应用网络策略，合理限制进入命名空间和 Pod 的流量。 + +:::note +这些是 CIS Benchmark 中的手动检查。CIS 扫描结果将标记为 `warning`，因为需要集群操作员进行手动检查。 +::: + +网络策略可以放置在 `/var/lib/rancher/k3s/server/manifests` 目录下的 `policy.yaml` 文件中。如果该目录不是作为 PSP（如上所述）的一部分创建的，则必须首先创建该目录。 + +```shell +sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/manifests +``` + +以下是合规的网络策略示例： + +```yaml +--- +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: intra-namespace + namespace: kube-system +spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + name: kube-system +--- +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: intra-namespace + namespace: default +spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + name: default +--- +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: intra-namespace + namespace: kube-public +spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + name: kube-public +``` + +除非特意允许，否则活动限制会阻止 DNS。以下是允许 DNS 相关流量的网络策略示例： + +```yaml +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-network-dns-policy + namespace: +spec: + ingress: + - ports: + - port: 53 + protocol: TCP + - port: 53 + protocol: UDP + podSelector: + matchLabels: + k8s-app: kube-dns + policyTypes: + - Ingress +``` + +如果没有创建网络策略来允许访问，则默认情况下会阻止 metrics-server 和 Traefik Ingress 控制器。 + +```yaml +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-all-metrics-server + namespace: kube-system +spec: + podSelector: + matchLabels: + k8s-app: metrics-server + ingress: + - {} + policyTypes: + - Ingress +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-all-svclbtraefik-ingress + namespace: kube-system +spec: + podSelector: + matchLabels: + svccontroller.k3s.cattle.io/svcname: traefik + ingress: + - {} + policyTypes: + - Ingress +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-all-traefik-v121-ingress + namespace: kube-system +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: traefik + ingress: + - {} + policyTypes: + - Ingress +``` + +:::note +你必须像平常一样管理你创建的任何其他命名空间的网络策略。 +::: + +### API server 审计配置 + +CIS 要求 1.2.19 至 1.2.22 与配置 API server 审核日志相关。默认情况下，K3s 不会创建日志目录和审计策略，因为每个用户的审计策略要求和环境都是特定的。 + +如果你需要日志目录，则必须在启动 K3s 之前创建它。我们建议限制访问权限以避免泄露敏感信息。 + +```bash +sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/logs +``` + +以下是用于记录请求元数据的初始审计策略。应将策略写入到 `/var/lib/rancher/k3s/server` 目录下名为 `audit.yaml` 的文件中。有关 API server 的策略配置的详细信息，请参阅 [官方 Kubernetes 文档](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/)。 + +```yaml +--- +apiVersion: audit.k8s.io/v1 +kind: Policy +rules: +- level: Metadata +``` + +还需要进一步配置才能通过 CIS 检查。这些在 K3s 中默认不配置，因为它们根据你的环境和需求而有所不同： + +- 确保 `--audit-log-path` 参数已经设置。 +- 确保 `--audit-log-maxage` 参数设置为 30 或适当的值。 +- 确保 `--audit-log-maxbackup` 参数设置为 10 或适当的值。 +- 确保 `--audit-log-maxsize` 参数设置为 100 或适当的值。 + +综合起来，要启用和配置审计日志，请将以下行添加到 Rancher 的 K3s 集群配置文件中： + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS 3.2.1 + - audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS 1.2.18 + - audit-log-maxage=30 # CIS 1.2.19 + - audit-log-maxbackup=10 # CIS 1.2.20 + - audit-log-maxsize=100 # CIS 1.2.21 +``` + +### Controller Manager 要求 + +CIS 要求 1.3.1 检查 Controller Manager 中的垃圾收集设置。垃圾收集对于确保资源充足可用性并避免性能和可用性下降非常重要。根据你的系统资源和测试结果，选择一个适当的阈值来激活垃圾收集。 + +你可以在 Rancher 的 K3s 集群文件中设置以下配置来解决此问题。下面的值仅是一个示例，请根据当前环境设置适当的阈值。 + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + kube-controller-manager-arg: + - terminated-pod-gc-threshold=10 # CIS 1.3.1 +``` + +### 配置 `default` Service Account + +Kubernetes 提供了一个名为 `default` 的 service account，供集群工作负载使用，其中没有为 Pod 分配特定的 service account。当 Pod 需要从 Kubernetes API 获取访问权限时，应为该 Pod 创建一个特定的 service account，并为该 service account 授予权限。 + +对于 CIS 5.1.5，`default` service account 应配置为不提供 service account 令牌，并且不具有任何明确的权限分配。 + +可以通过在每个命名空间中将 `default` service account 的 `automountServiceAccountToken` 字段更新为 `false` 来解决此问题。 + +对于内置命名空间（`kube-system`、`kube-public`、`kube-node-lease` 和 `default`）中的 `default` service accounts，K3s 不会自动执行此操作。 + +将以下配置保存到名为 `account_update.yaml` 的文件中。 + +```yaml +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: default +automountServiceAccountToken: false +``` + +创建一个名为 `account_update.sh` 的 Bash 脚本文件。确保使用 `chmod +x account_update.sh` 给脚本添加可执行权限。 + +```shell +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" +done +``` + +每次向你的集群添加新的 service account 时，运行该脚本。 + +## 加固版 K3s 模板配置参考 + +Rancher 使用以下参考模板配置，基于本指南中的每个 CIS 控件创建加固过的自定义 K3s 集群。此参考内容不包括其他必需的**集群配置**指令，这些指令因你的环境而异。 + + + + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: # 定义集群名称 +spec: + defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted + enableNetworkPolicy: true + kubernetesVersion: # 定义 K3s 版本 + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - enable-admission-plugins=NodeRestriction,ServiceAccount # CIS 1.2.15, 1.2.13 + - audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS 3.2.1 + - audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS 1.2.18 + - audit-log-maxage=30 # CIS 1.2.19 + - audit-log-maxbackup=10 # CIS 1.2.20 + - audit-log-maxsize=100 # CIS 1.2.21 + - request-timeout=300s # CIS 1.2.22 + - service-account-lookup=true # CIS 1.2.24 + kube-controller-manager-arg: + - terminated-pod-gc-threshold=10 # CIS 1.3.1 + secrets-encryption: true + machineSelectorConfig: + - config: + kubelet-arg: + - make-iptables-util-chains=true # CIS 4.2.7 +``` + + + + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: # 定义集群名称 +spec: + enableNetworkPolicy: true + kubernetesVersion: # 定义 K3s 版本 + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount # CIS 1.2.15, 5.2, 1.2.13 + - audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS 3.2.1 + - audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS 1.2.18 + - audit-log-maxage=30 # CIS 1.2.19 + - audit-log-maxbackup=10 # CIS 1.2.20 + - audit-log-maxsize=100 # CIS 1.2.21 + - request-timeout=300s # CIS 1.2.22 + - service-account-lookup=true # CIS 1.2.24 + kube-controller-manager-arg: + - terminated-pod-gc-threshold=10 # CIS 1.3.1 + secrets-encryption: true + machineSelectorConfig: + - config: + kubelet-arg: + - make-iptables-util-chains=true # CIS 4.2.7 + protect-kernel-defaults: true # CIS 4.2.6 +``` + + + + +## 结论 + +如果你按照本指南操作，由 Rancher 提供的 K3s 自定义集群将配置为通过 CIS Kubernetes Benchmark 测试。你可以查看我们的 K3s 自我评估指南，了解我们是如何验证每个 benchmarks 的，并且你可以在你的集群上执行相同的操作。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md index ddf9809440a..5bfb97c99f7 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md @@ -1,36 +1,40 @@ --- -title: K3s Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.23 +title: K3s 自我评估指南 - CIS Benchmark v1.23 - K8s v1.23 --- -This document is a companion to the [K3s Hardening Guide](../../../../pages-for-subheaders/k3s-hardening-guide.md), which provides prescriptive guidance on how to harden K3s clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. + + + -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: +本文档是 [K3s 加固指南](k3s-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 K3s 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 -| Rancher Version | CIS Benchmark Version | Kubernetes Version | +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | |-----------------|-----------------------|--------------------| | Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | -This document is for Rancher operators, security teams, auditors and decision makers. +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.23 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 -## Testing Methodology +## 测试方法 -Each control in the CIS Kubernetes Benchmark was evaluated against a K3s cluster that was configured according to the accompanying hardening guide. +每个 CIS Kubernetes Benchmark 中的 control 都根据附带的加固指南评估了针对 K3s 集群的配置。 -Where control audits differ from the original CIS benchmark, the audit commands specific to K3s are provided for testing. +当 control 审计与原始的 CIS benchmark 不同的时候，提供了针对 K3s 的特定审计命令，以供测试使用。 -These are the possible results for each control: +以下是每个 control 可能的结果： -- **Pass** - The K3s cluster passes the audit outlined in the benchmark. -- **Not Applicable** - The control is not applicable to K3s because of how it is designed to operate. The remediation section explains why. -- **Warn** - The control is manual in the CIS benchmark and it depends on the cluster's use-case or some other factor that must be determined by the cluster operator. These controls have been evaluated to ensure K3s doesn't prevent their implementation, but no further configuration or auditing of the cluster has been performed. +- **Pass（通过）** - K3s 集群通过了 benchmark 中概述的审计。 +- **Not Applicable（不适用）** - 由于 K3s 的设计方式，该 control 不适用于 K3s。在补救措施部分解释了原因。 +- **Warn（警告）** - 在 CIS benchmark 中，该 control 是手动的，它取决于集群的使用情况或其他必须由集群操作员确定的因素。这些 control 措施已经过评估，以确保 K3s 不会阻止其实施，但尚未对集群进行进一步的配置或审计。 -This guide makes the assumption that K3s is running as a Systemd unit. Your installation may vary. Adjust the "audit" commands to fit your scenario. +本指南假设 K3s 作为 Systemd 单元运行。你的安装可能会有所不同。调整"审计"命令以适合你的场景。 :::note -This guide only covers `automated` (previously called `scored`) tests. +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md deleted file mode 100644 index 8944de04c27..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md +++ /dev/null @@ -1,3148 +0,0 @@ ---- -title: K3s Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.24 ---- - -This document is a companion to the [K3s Hardening Guide](../../../../pages-for-subheaders/k3s-hardening-guide.md), which provides prescriptive guidance on how to harden K3s clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.24 | - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -Each control in the CIS Kubernetes Benchmark was evaluated against a K3s cluster that was configured according to the accompanying hardening guide. - -Where control audits differ from the original CIS benchmark, the audit commands specific to K3s are provided for testing. - -These are the possible results for each control: - -- **Pass** - The K3s cluster passes the audit outlined in the benchmark. -- **Not Applicable** - The control is not applicable to K3s because of how it is designed to operate. The remediation section explains why. -- **Warn** - The control is manual in the CIS benchmark and it depends on the cluster's use-case or some other factor that must be determined by the cluster operator. These controls have been evaluated to ensure K3s doesn't prevent their implementation, but no further configuration or auditing of the cluster has been performed. - -This guide makes the assumption that K3s is running as a Systemd unit. Your installation may vary. Adjust the "audit" commands to fit your scenario. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Control Plane Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the -control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-apiserver.yaml - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-controller-manager.yaml - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-scheduler.yaml - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 /etc/kubernetes/manifests/etcd.yaml - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /etc/kubernetes/manifests/etcd.yaml - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 1.1.11 -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** Not Applicable - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/admin.conf - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 scheduler - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root scheduler - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 controllermanager - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/controller.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root controllermanager - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown -R root:root /etc/kubernetes/pki/ - -**Audit:** - -```bash -find /var/lib/rancher/k3s/server/tls | xargs stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) - - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - -**Audit:** - -```bash -stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) - - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /etc/kubernetes/pki/*.key - -**Audit:** - -```bash -stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep containerd | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' is present OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 521 1 0 22:09 ? 00:00:00 /usr/bin/containerd root 802 1 0 22:09 ? 00:00:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 3970 1 0 22:13 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id 836d5508918bf22689e12f2885aeb9de5c2420beb956f6dc587d11b049e4edf4 -address /run/k3s/containerd/containerd.sock root 4088 1 0 22:13 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id 6148f976b57db791da86119c599700f683df838df56b1382d428ec38402caa64 -address /run/k3s/containerd/containerd.sock root 4109 1 0 22:13 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id b0e22b39e80606c4ad5d7fdafe21a771a27a5dce8f06155af579a9fad8158219 -address /run/k3s/containerd/containerd.sock root 5321 1 0 22:14 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id f60b96ef6beaf1b7903aba944af101a7d401082969ad8b5aea633bda1043b165 -address /run/k3s/containerd/containerd.sock root 5395 1 0 22:14 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id df001d7d85239f1cfdd8e041db801a6383bc18e5e10d0cd8e8e4e6ca9b7ce036 -address /run/k3s/containerd/containerd.sock root 6366 1 0 22:15 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id a900df9f736ca35b5dbd603034588a461d706ae6c83529136a81ec13e0ecedc0 -address /run/k3s/containerd/containerd.sock root 7859 1 0 22:15 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id b3c1510cafad33b31758db1b7c771efe98f0d395d5d52caf90db7bd379966299 -address /run/k3s/containerd/containerd.sock root 7896 1 0 22:15 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id e5720fc8f9e95f2fd8b8e87c93f25213315d78b630fc889b2a5cb8b18f2a1442 -address /run/k3s/containerd/containerd.sock root 9773 9761 4 22:22 ? 00:00:16 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 10275 1 0 22:22 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id 8b29d2205c5f7d47d0ecd4f95b7b96062454233e66450f9163058647e7b5c390 -address /run/k3s/containerd/containerd.sock root 11185 1 0 22:23 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9581871a145dd1c0bbff06cddea0030a30de2760e90c5fa9154cccb250f05faa -address /run/k3s/containerd/containerd.sock root 13605 1 0 22:27 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id e05825dc2a33d576c35cc1b56610d3e4459d209524645f95f163c100f6e5df46 -address /run/k3s/containerd/containerd.sock root 13770 1 0 22:27 ? 00:00:00 /var/lib/rancher/k3s/data/4cdfcad9f220e885cbc32cf86c6cb0d26b496e3949efb0aa33fb37692e11d521/bin/containerd-shim-runc-v2 -namespace k8s.io -id 74d10151f05f78b0d38deb2b1ee045bf7b72e7b879e6a15112c8495f98d230b8 -address /run/k3s/containerd/containerd.sock -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'AlwaysPullImages' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'ServiceAccount' -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-path' -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxage' -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxbackup' -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxsize' -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.23 Ensure that the --request-timeout argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameter as appropriate and if needed. -For example, --request-timeout=300s - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'request-timeout' -``` - -**Expected Result**: - -```console -'--request-timeout' is not present OR '--request-timeout' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-lookup' -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.25 Ensure that the --service-account-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-key-file' -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 1.2.29 -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'encryption-provider-config' -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.31 Ensure that encryption providers are appropriately configured (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit:** - -```bash -grep aescbc /path/to/encryption-config.json -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) - - -**Result:** warn - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'bind-address' -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.1 -``` - -**Expected Result**: - -```console -'cert-file' is present AND 'key-file' is present -``` - -**Returned Value**: - -```console -cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key -``` - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and set the below parameter. ---client-cert-auth="true" - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.2 -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -client-cert-auth: true -``` - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.3 -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.4 -``` - -**Expected Result**: - -```console -'cert-file' is present AND 'key-file' is present -``` - -**Returned Value**: - -```console -cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key -``` - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and set the below parameter. ---peer-client-cert-auth=true - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.5 -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -client-cert-auth: true -``` - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.6 -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.7 -``` - -**Expected Result**: - -```console -'trusted-ca-file' is present -``` - -**Returned Value**: - -```console -trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Manual) - - -**Result:** warn - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig -``` - -**Expected Result**: - -```console -'permissions' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/agent/kubelet.kubeconfig -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt -``` - -**Expected Result**: - -```console -'644' is equal to '644' OR '640' is present OR '600' is present OR '444' is present OR '440' is present OR '400' is present OR '000' is present -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" | grep -v grep; else echo "--anonymous-auth=false"; fi' -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" | grep -v grep; else echo "--authorization-mode=Webhook"; fi' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" | grep -v grep; else echo "--client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt"; fi' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:02 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:02Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:03 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:03Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-2 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=995abdc2-f967-4bd6-936e-f3c26594573c --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'protect-kernel-defaults' -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -Feb 27 22:22:03 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:03Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-2 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=995abdc2-f967-4bd6-936e-f3c26594573c --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' -``` - -**Expected Result**: - -```console -'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -**Returned Value**: - -```console -Feb 27 22:22:03 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:03Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-2 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=995abdc2-f967-4bd6-936e-f3c26594573c --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 22:22:03 ip-172-31-25-2 k3s[9761]: time="2023-02-27T22:22:03Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-2 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=995abdc2-f967-4bd6-936e-f3c26594573c --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Manual) - - -**Result:** warn - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** warn - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) - - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md deleted file mode 100644 index 1f5b7dfd37b..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md +++ /dev/null @@ -1,3148 +0,0 @@ ---- -title: K3s Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.25 ---- - -This document is a companion to the [K3s Hardening Guide](../../../../pages-for-subheaders/k3s-hardening-guide.md), which provides prescriptive guidance on how to harden K3s clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.25 | - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -Each control in the CIS Kubernetes Benchmark was evaluated against a K3s cluster that was configured according to the accompanying hardening guide. - -Where control audits differ from the original CIS benchmark, the audit commands specific to K3s are provided for testing. - -These are the possible results for each control: - -- **Pass** - The K3s cluster passes the audit outlined in the benchmark. -- **Not Applicable** - The control is not applicable to K3s because of how it is designed to operate. The remediation section explains why. -- **Warn** - The control is manual in the CIS benchmark and it depends on the cluster's use-case or some other factor that must be determined by the cluster operator. These controls have been evaluated to ensure K3s doesn't prevent their implementation, but no further configuration or auditing of the cluster has been performed. - -This guide makes the assumption that K3s is running as a Systemd unit. Your installation may vary. Adjust the "audit" commands to fit your scenario. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Control Plane Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the -control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-apiserver.yaml - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-controller-manager.yaml - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-scheduler.yaml - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 /etc/kubernetes/manifests/etcd.yaml - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /etc/kubernetes/manifests/etcd.yaml - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 1.1.11 -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** Not Applicable - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/admin.conf - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 scheduler - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root scheduler - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 controllermanager - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/controller.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root controllermanager - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown -R root:root /etc/kubernetes/pki/ - -**Audit:** - -```bash -find /var/lib/rancher/k3s/server/tls | xargs stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) - - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - -**Audit:** - -```bash -stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) - - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /etc/kubernetes/pki/*.key - -**Audit:** - -```bash -stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep containerd | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' is present OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 519 1 0 22:09 ? 00:00:00 /usr/bin/containerd root 801 1 0 22:09 ? 00:00:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 3864 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id d00174abbc275f6bb85c7f0be1d3154b9c91982a10b9dba6b5cb280f4d4c531d -address /run/k3s/containerd/containerd.sock root 4105 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7c2b546b4d2380bcb51278661f34cff94fad2ba06978e13f8f1b92dafcc89d43 -address /run/k3s/containerd/containerd.sock root 4206 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 68d8a55ff4663985be004608dbf78b0362f5522e18490c81d4c8dc9963de1556 -address /run/k3s/containerd/containerd.sock root 5374 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id ca0ae9e0b37dfd7b1ce05f72e1bc5a1be8f5cb08f2b4543081536de3bdbc925d -address /run/k3s/containerd/containerd.sock root 5443 1 0 22:31 ? 00:00:01 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3ea3c1cdbbd5adb8efd5c67a46aadd0fca9918dc0ad1f7cafe38b83171e3dc1b -address /run/k3s/containerd/containerd.sock root 7130 1 0 22:32 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4d838297d35a31003106ac5989c3547433985bb2964b47baad12cee6e375645e -address /run/k3s/containerd/containerd.sock root 7639 1 0 22:32 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 341cb9bcd8486aa2f1acb8e1ae51baebd630ac6ed266643266c34d677f61c7d0 -address /run/k3s/containerd/containerd.sock root 10308 1 0 23:17 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id c534fbee8e0d06fd9b29bf8fc70a138975c6b18db25f1faf2615677dfdb4199e -address /run/k3s/containerd/containerd.sock root 11370 1 0 23:18 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4ff4b8776dac7a35b83616d341dbe4d5a689ac7fb9b8eee8db5978e3968380ea -address /run/k3s/containerd/containerd.sock root 13736 13723 2 23:21 ? 00:00:10 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 16022 1 0 23:29 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9027256349086e458119478e5e00384b1b76fbf5e6dbee23699f596a88d9f2bc -address /run/k3s/containerd/containerd.sock root 16159 1 0 23:29 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 929bf369fc5881654f4c1925624151ddb7cea51073267b8d213d966ba45406f3 -address /run/k3s/containerd/containerd.sock -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'AlwaysPullImages' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'ServiceAccount' -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-path' -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxage' -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxbackup' -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxsize' -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.23 Ensure that the --request-timeout argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameter as appropriate and if needed. -For example, --request-timeout=300s - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'request-timeout' -``` - -**Expected Result**: - -```console -'--request-timeout' is not present OR '--request-timeout' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-lookup' -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.25 Ensure that the --service-account-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-key-file' -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 1.2.29 -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'encryption-provider-config' -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.31 Ensure that encryption providers are appropriately configured (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit:** - -```bash -grep aescbc /path/to/encryption-config.json -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) - - -**Result:** warn - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'bind-address' -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.1 -``` - -**Expected Result**: - -```console -'cert-file' is present AND 'key-file' is present -``` - -**Returned Value**: - -```console -cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key -``` - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and set the below parameter. ---client-cert-auth="true" - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.2 -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -client-cert-auth: true -``` - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.3 -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.4 -``` - -**Expected Result**: - -```console -'cert-file' is present AND 'key-file' is present -``` - -**Returned Value**: - -```console -cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key -``` - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and set the below parameter. ---peer-client-cert-auth=true - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.5 -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -client-cert-auth: true -``` - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.6 -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.7 -``` - -**Expected Result**: - -```console -'trusted-ca-file' is present -``` - -**Returned Value**: - -```console -trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Manual) - - -**Result:** warn - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig -``` - -**Expected Result**: - -```console -'permissions' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/agent/kubelet.kubeconfig -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt -``` - -**Expected Result**: - -```console -'644' is equal to '644' OR '640' is present OR '600' is present OR '444' is present OR '440' is present OR '400' is present OR '000' is present -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" | grep -v grep; else echo "--anonymous-auth=false"; fi' -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" | grep -v grep; else echo "--authorization-mode=Webhook"; fi' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" | grep -v grep; else echo "--client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt"; fi' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'protect-kernel-defaults' -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' -``` - -**Expected Result**: - -```console -'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Manual) - - -**Result:** warn - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** warn - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) - - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md new file mode 100644 index 00000000000..464073a825f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md @@ -0,0 +1,3208 @@ +--- +title: K3s 自我评估指南 - CIS Benchmark v1.24 - K8s v1.24 +--- + + + + + +本文档是 [K3s 加固指南](k3s-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 K3s 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.24 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +每个 CIS Kubernetes Benchmark 中的 control 都根据附带的加固指南评估了针对 K3s 集群的配置。 + +当 control 审计与原始的 CIS benchmark 不同的时候，提供了针对 K3s 的特定审计命令，以供测试使用。 + +以下是每个 control 可能的结果： + +- **Pass（通过）** - K3s 集群通过了 benchmark 中概述的审计。 +- **Not Applicable（不适用）** - 由于 K3s 的设计方式，该 control 不适用于 K3s。在补救措施部分解释了原因。 +- **Warn（警告）** - 在 CIS benchmark 中，该 control 是手动的，它取决于集群的使用情况或其他必须由集群操作员确定的因素。这些 control 措施已经过评估，以确保 K3s 不会阻止其实施，但尚未对集群进行进一步的配置或审计。 + +本指南假设 K3s 作为 Systemd 单元运行。你的安装可能会有所不同。调整"审计"命令以适合你的场景。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 644 /etc/kubernetes/manifests/kube-apiserver.yaml + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-controller-manager.yaml + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-scheduler.yaml + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /etc/kubernetes/manifests/etcd.yaml + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /etc/kubernetes/manifests/etcd.yaml + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.1.11 +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** Not Applicable + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 scheduler + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root scheduler + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 controllermanager + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/controller.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root controllermanager + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/tls +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit:** + +```bash +find /var/lib/rancher/k3s/server/tls | xargs stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.crt + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.key + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' is present OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 410 1 0 Sep11 ? 00:01:50 /usr/bin/containerd root 539 1 0 Sep11 ? 00:00:09 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 45213 45195 3 Sep11 ? 00:45:14 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 47188 1 0 Sep11 ? 00:01:00 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 02b8fdd94b7628d575ace92f337dcc93202151d07f33858341cfcb0178fea586 -address /run/k3s/containerd/containerd.sock root 47235 1 0 Sep11 ? 00:00:33 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id eb0f03c4bc1125ae34e5de77c7eaeb9c2c1e46bec0bb106b99281df7353f9ded -address /run/k3s/containerd/containerd.sock root 47948 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5eece7110bd3876414f890ed1b7158eb91f60d1f14450a9f8c77a0fc62b73c4a -address /run/k3s/containerd/containerd.sock root 48047 1 0 Sep11 ? 00:00:32 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4ef831f4bb8830a0a7b46a1e8be58197be00846ea0caa094014ea7ce3adf008e -address /run/k3s/containerd/containerd.sock root 48220 1 0 Sep11 ? 00:00:33 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id f8d5d3791ad2b1a8534476aa2fbdf8da1e4f0b9fdcdd26b1ae161142458a5336 -address /run/k3s/containerd/containerd.sock root 48878 1 0 Sep11 ? 00:00:32 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 41b93a6581e272b0852257b3d7c44ab4cadcdf465a3579a8b29b493ea75feaf7 -address /run/k3s/containerd/containerd.sock root 49870 1 0 Sep11 ? 00:00:30 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id be329cdccda4049f048c759a2b1915cd72526473ff7768e895bcfc153a66c125 -address /run/k3s/containerd/containerd.sock root 50271 1 0 Sep11 ? 00:00:32 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id cceb5d26eba7d7d3c474a2dc842b72495689f4e40055505ea2598e4ac6849d40 -address /run/k3s/containerd/containerd.sock root 50571 1 0 Sep11 ? 00:00:32 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3a26e691e0d0186165d523bef22b0af71f175c39df82845b32ce7b4e3d652d4a -address /run/k3s/containerd/containerd.sock root 96693 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9ed77e6e58410f1295f730ba4a8f8daf7aff8f4e896e24219a8fa710729c2287 -address /run/k3s/containerd/containerd.sock root 97726 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 1be4fe50e6bb29d4549f8b03102b5dc15b7669fe56b32beae3f701cf6d2e58cc -address /run/k3s/containerd/containerd.sock root 97923 1 1 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7c65c47376f36a8980a5bebad7ffdc1abe1853821cf6c4a1d45f1b6be3452062 -address /run/k3s/containerd/containerd.sock +``` + +### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --kubelet-https parameter. + +### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'ServiceAccount' +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.18 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-path' +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxage' +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxbackup' +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxsize' +``` + +**Expected Result**: + +```console +'--audit-log-maxsize' is greater or equal to 100 +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.23 Ensure that the --request-timeout argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, --request-timeout=300s + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'request-timeout' +``` + +**Expected Result**: + +```console +'--request-timeout' is not present OR '--request-timeout' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-lookup' +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.25 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-key-file' +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.2.29 +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'encryption-provider-config' +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.31 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit:** + +```bash +grep aescbc /path/to/encryption-config.json +``` + +### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'bind-address' +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.1 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.2 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.3 +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.4 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.5 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.6 +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.7 +``` + +**Expected Result**: + +```console +'trusted-ca-file' is present +``` + +**Returned Value**: + +```console +trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Manual) + + +**Result:** warn + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +**Returned Value**: + +```console +600 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'600' is equal to '600' +``` + +**Returned Value**: + +```console +600 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +**Returned Value**: + +```console +644 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/kubelet/config.yaml + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" | grep -v grep; else echo "--anonymous-auth=false"; fi' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" | grep -v grep; else echo "--authorization-mode=Webhook"; fi' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" | grep -v grep; else echo "--client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt"; fi' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:32:02 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:32:02Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-3-32 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=cc5faddd-9c87-4a04-97c2-dcf4cbcbe10f --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' +``` + +### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'protect-kernel-defaults' +``` + +**Expected Result**: + +```console +'--protect-kernel-defaults' is equal to 'true' +``` + +**Returned Value**: + +```console +Sep 11 20:32:02 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:32:02Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-3-32 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=cc5faddd-9c87-4a04-97c2-dcf4cbcbe10f --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' +``` + +**Expected Result**: + +```console +'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:32:02 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:32:02Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-3-32 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=cc5faddd-9c87-4a04-97c2-dcf4cbcbe10f --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +### 4.2.9 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'--event-qps' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 410 1 0 Sep11 ? 00:01:50 /usr/bin/containerd root 45213 45195 3 Sep11 ? 00:45:15 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:32:02 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:32:02Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-3-32 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=cc5faddd-9c87-4a04-97c2-dcf4cbcbe10f --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 410 1 0 Sep11 ? 00:01:50 /usr/bin/containerd root 45213 45195 3 Sep11 ? 00:45:15 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 410 1 0 Sep11 ? 00:01:50 /usr/bin/containerd root 45213 45195 3 Sep11 ? 00:45:15 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** warn + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +### 5.2.7 Minimize the admission of root containers (Automated) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Automated) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md new file mode 100644 index 00000000000..8443a80b530 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md @@ -0,0 +1,3215 @@ +--- +title: K3s 自我评估指南 - CIS Benchmark v1.7 - K8s v1.25/v1.26/v1.27 +--- + + + + + +本文档是 [K3s 加固指南](k3s-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 K3s 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25/v1.26/v1.27 | + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.7 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +每个 CIS Kubernetes Benchmark 中的 control 都根据附带的加固指南评估了针对 K3s 集群的配置。 + +当 control 审计与原始的 CIS benchmark 不同的时候，提供了针对 K3s 的特定审计命令，以供测试使用。 + +以下是每个 control 可能的结果： + +- **Pass（通过）** - K3s 集群通过了 benchmark 中概述的审计。 +- **Not Applicable（不适用）** - 由于 K3s 的设计方式，该 control 不适用于 K3s。在补救措施部分解释了原因。 +- **Warn（警告）** - 在 CIS benchmark 中，该 control 是手动的，它取决于集群的使用情况或其他必须由集群操作员确定的因素。这些 control 措施已经过评估，以确保 K3s 不会阻止其实施，但尚未对集群进行进一步的配置或审计。 + +本指南假设 K3s 作为 Systemd 单元运行。你的安装可能会有所不同。调整"审计"命令以适合你的场景。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-apiserver.yaml +Not Applicable. + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml +Not Applicable. + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-controller-manager.yaml +Not Applicable. + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml +Not Applicable. + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-scheduler.yaml +Not Applicable. + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml +Not Applicable. + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /etc/kubernetes/manifests/etcd.yaml +Not Applicable. + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /etc/kubernetes/manifests/etcd.yaml +Not Applicable. + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 +Not Applicable. + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root +Not Applicable. + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.1.11 +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** Not Applicable + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd +Not Applicable. + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 scheduler + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root scheduler + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 controllermanager + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/controller.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root controllermanager + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/cred/controller.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /var/lib/rancher/k3s/server/tls + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/tls +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.crt + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.key + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 663 1 0 Sep11 ? 00:00:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 3021 1 0 Sep11 ? 00:00:29 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4790d392966915d995e666002c56ed4cce6dd86f305ce2ee390547a1fcbf6c82 -address /run/k3s/containerd/containerd.sock root 3035 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9d73e6a160ccde6c7c7d4ba0df3b9f696e3de2ebfc0f19a1dcbdf13aea496427 -address /run/k3s/containerd/containerd.sock root 3235 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 587f6221ee9f36c877231ada8d816a799dfda186332c33378d1eb16c72cdc87d -address /run/k3s/containerd/containerd.sock root 4435 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a74c9ec7d99785c2f2d4e6826aa80c22eb8b38249e8f99679ece00a818e9b7b3 -address /run/k3s/containerd/containerd.sock root 4985 1 0 Sep11 ? 00:00:53 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b0c9784dbe0fcbe8be10c857976e70ec84a208cb814e87b5ca085a02d434f8c -address /run/k3s/containerd/containerd.sock root 5056 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id f3231ff35f18056e74eda14907f296be15f7ea1c6ae5ab7904e27d4d18183301 -address /run/k3s/containerd/containerd.sock root 5868 1 0 Sep11 ? 00:00:27 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3e908c4d0b10df275bdef6f72fbcfa09517d11cf749236ad020364e14d77bc93 -address /run/k3s/containerd/containerd.sock root 6158 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id bec5780a5c73fa3154aa3b5ee26cdf23202db821205893e7e66ae17e6103e97b -address /run/k3s/containerd/containerd.sock root 7366 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a81b78845bdcaef710314c93e5ea0d0617f37a8929472f7b570ab90c6667f57f -address /run/k3s/containerd/containerd.sock root 97274 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id c90652c935d9e79af45b9a9ac2b4fe315e2a761a0525737fdb95c680123a164c -address /run/k3s/containerd/containerd.sock root 98309 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 431a2763636488efd7d104c228d41f4f2ecd8c06a7fc375d8977ab0d238936a8 -address /run/k3s/containerd/containerd.sock root 98493 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b480d2fa55c8cc0105ec1902887040c017f03d5b1eb6c73c24bb9d523ad9b37 -address /run/k3s/containerd/containerd.sock +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' is present OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 663 1 0 Sep11 ? 00:00:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 3021 1 0 Sep11 ? 00:00:29 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4790d392966915d995e666002c56ed4cce6dd86f305ce2ee390547a1fcbf6c82 -address /run/k3s/containerd/containerd.sock root 3035 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9d73e6a160ccde6c7c7d4ba0df3b9f696e3de2ebfc0f19a1dcbdf13aea496427 -address /run/k3s/containerd/containerd.sock root 3235 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 587f6221ee9f36c877231ada8d816a799dfda186332c33378d1eb16c72cdc87d -address /run/k3s/containerd/containerd.sock root 4435 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a74c9ec7d99785c2f2d4e6826aa80c22eb8b38249e8f99679ece00a818e9b7b3 -address /run/k3s/containerd/containerd.sock root 4985 1 0 Sep11 ? 00:00:53 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b0c9784dbe0fcbe8be10c857976e70ec84a208cb814e87b5ca085a02d434f8c -address /run/k3s/containerd/containerd.sock root 5056 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id f3231ff35f18056e74eda14907f296be15f7ea1c6ae5ab7904e27d4d18183301 -address /run/k3s/containerd/containerd.sock root 5868 1 0 Sep11 ? 00:00:27 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3e908c4d0b10df275bdef6f72fbcfa09517d11cf749236ad020364e14d77bc93 -address /run/k3s/containerd/containerd.sock root 6158 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id bec5780a5c73fa3154aa3b5ee26cdf23202db821205893e7e66ae17e6103e97b -address /run/k3s/containerd/containerd.sock root 7366 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a81b78845bdcaef710314c93e5ea0d0617f37a8929472f7b570ab90c6667f57f -address /run/k3s/containerd/containerd.sock root 97274 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id c90652c935d9e79af45b9a9ac2b4fe315e2a761a0525737fdb95c680123a164c -address /run/k3s/containerd/containerd.sock root 98309 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 431a2763636488efd7d104c228d41f4f2ecd8c06a7fc375d8977ab0d238936a8 -address /run/k3s/containerd/containerd.sock root 98493 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b480d2fa55c8cc0105ec1902887040c017f03d5b1eb6c73c24bb9d523ad9b37 -address /run/k3s/containerd/containerd.sock +``` + +### 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= +Permissive - When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. + +### 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.7 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.9 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' is present +``` + +**Returned Value**: + +```console +root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 663 1 0 Sep11 ? 00:00:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 3021 1 0 Sep11 ? 00:00:29 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4790d392966915d995e666002c56ed4cce6dd86f305ce2ee390547a1fcbf6c82 -address /run/k3s/containerd/containerd.sock root 3035 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9d73e6a160ccde6c7c7d4ba0df3b9f696e3de2ebfc0f19a1dcbdf13aea496427 -address /run/k3s/containerd/containerd.sock root 3235 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 587f6221ee9f36c877231ada8d816a799dfda186332c33378d1eb16c72cdc87d -address /run/k3s/containerd/containerd.sock root 4435 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a74c9ec7d99785c2f2d4e6826aa80c22eb8b38249e8f99679ece00a818e9b7b3 -address /run/k3s/containerd/containerd.sock root 4985 1 0 Sep11 ? 00:00:53 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b0c9784dbe0fcbe8be10c857976e70ec84a208cb814e87b5ca085a02d434f8c -address /run/k3s/containerd/containerd.sock root 5056 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id f3231ff35f18056e74eda14907f296be15f7ea1c6ae5ab7904e27d4d18183301 -address /run/k3s/containerd/containerd.sock root 5868 1 0 Sep11 ? 00:00:27 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3e908c4d0b10df275bdef6f72fbcfa09517d11cf749236ad020364e14d77bc93 -address /run/k3s/containerd/containerd.sock root 6158 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id bec5780a5c73fa3154aa3b5ee26cdf23202db821205893e7e66ae17e6103e97b -address /run/k3s/containerd/containerd.sock root 7366 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a81b78845bdcaef710314c93e5ea0d0617f37a8929472f7b570ab90c6667f57f -address /run/k3s/containerd/containerd.sock root 97274 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id c90652c935d9e79af45b9a9ac2b4fe315e2a761a0525737fdb95c680123a164c -address /run/k3s/containerd/containerd.sock root 98309 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 431a2763636488efd7d104c228d41f4f2ecd8c06a7fc375d8977ab0d238936a8 -address /run/k3s/containerd/containerd.sock root 98493 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b480d2fa55c8cc0105ec1902887040c017f03d5b1eb6c73c24bb9d523ad9b37 -address /run/k3s/containerd/containerd.sock +``` + +### 1.2.12 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... +Permissive - Enabling Pod Security Policy can cause applications to unexpectedly fail. + +### 1.2.13 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.14 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.15 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.16 Ensure that the --secure-port argument is not set to 0 - NoteThis recommendation is obsolete and will be deleted per the consensus process (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.17 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.18 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log +Permissive. + +### 1.2.19 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 +Permissive. + +### 1.2.20 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 +Permissive. + +### 1.2.21 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 +Permissive. + +### 1.2.22 Ensure that the --request-timeout argument is set as appropriate (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, --request-timeout=300s +Permissive. + +### 1.2.23 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.24 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +### 1.2.25 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.2.29 +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.26 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +### 1.2.27 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.28 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.29 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= +Permissive - Enabling encryption changes how data can be recovered as data is encrypted. + +### 1.2.30 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. +Permissive - Enabling encryption changes how data can be recovered as data is encrypted. + +### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true +Not Applicable. + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 663 1 0 Sep11 ? 00:00:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 3021 1 0 Sep11 ? 00:00:29 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4790d392966915d995e666002c56ed4cce6dd86f305ce2ee390547a1fcbf6c82 -address /run/k3s/containerd/containerd.sock root 3035 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9d73e6a160ccde6c7c7d4ba0df3b9f696e3de2ebfc0f19a1dcbdf13aea496427 -address /run/k3s/containerd/containerd.sock root 3235 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 587f6221ee9f36c877231ada8d816a799dfda186332c33378d1eb16c72cdc87d -address /run/k3s/containerd/containerd.sock root 4435 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a74c9ec7d99785c2f2d4e6826aa80c22eb8b38249e8f99679ece00a818e9b7b3 -address /run/k3s/containerd/containerd.sock root 4985 1 0 Sep11 ? 00:00:53 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b0c9784dbe0fcbe8be10c857976e70ec84a208cb814e87b5ca085a02d434f8c -address /run/k3s/containerd/containerd.sock root 5056 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id f3231ff35f18056e74eda14907f296be15f7ea1c6ae5ab7904e27d4d18183301 -address /run/k3s/containerd/containerd.sock root 5868 1 0 Sep11 ? 00:00:27 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3e908c4d0b10df275bdef6f72fbcfa09517d11cf749236ad020364e14d77bc93 -address /run/k3s/containerd/containerd.sock root 6158 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id bec5780a5c73fa3154aa3b5ee26cdf23202db821205893e7e66ae17e6103e97b -address /run/k3s/containerd/containerd.sock root 7366 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a81b78845bdcaef710314c93e5ea0d0617f37a8929472f7b570ab90c6667f57f -address /run/k3s/containerd/containerd.sock root 97274 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id c90652c935d9e79af45b9a9ac2b4fe315e2a761a0525737fdb95c680123a164c -address /run/k3s/containerd/containerd.sock root 98309 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 431a2763636488efd7d104c228d41f4f2ecd8c06a7fc375d8977ab0d238936a8 -address /run/k3s/containerd/containerd.sock root 98493 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b480d2fa55c8cc0105ec1902887040c017f03d5b1eb6c73c24bb9d523ad9b37 -address /run/k3s/containerd/containerd.sock +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.1 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.2 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.3 +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.4 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.5 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.6 +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.7 +``` + +**Expected Result**: + +```console +'trusted-ca-file' is present +``` + +**Returned Value**: + +```console +trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +### 3.1.2 Service account token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of service account tokens. + +### 3.1.3 Bootstrap token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of bootstrap tokens. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Manual) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable - All configuration is passed in as arguments at container run time. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable. + All configuration is passed in as arguments at container run time. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +**Returned Value**: + +```console +600 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +**Returned Value**: + +```console +644 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/kubelet/config.yaml + +### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml +Not Applicable. +All configuration is passed in as arguments at container run time. + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" | grep -v grep; else echo "--anonymous-auth=false"; fi' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" | grep -v grep; else echo "--authorization-mode=Webhook"; fi' +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" | grep -v grep; else echo "--client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt"; fi' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:15 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:15Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-12-34 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=af02ecbc-1e4e-422e-8b4d-4b2aa24a9d46 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' +``` + +### 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' +``` + +**Expected Result**: + +```console +'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:15 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:15Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-12-34 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=af02ecbc-1e4e-422e-8b4d-4b2aa24a9d46 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.7 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable. + +### 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--event-qps' is present OR '--event-qps' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** Not Applicable + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service +Permissive - When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. + +### 4.2.10 Ensure that the --rotate-certificates argument is not set to false (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** pass + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable. + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'RotateKubeletServerCertificate' is present OR 'RotateKubeletServerCertificate' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.13 Ensure that a limit is set on pod PIDs (Manual) + + +**Result:** warn + +**Remediation:** +Decide on an appropriate level for this parameter and set it, +either via the --pod-max-pids command line parameter or the PodPidsLimit configuration file setting. + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--pod-max-pids' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[]?.kind=="ServiceAccount" and .subjects[]?.name=="default") or (.subjects[]?.kind=="Group" and .subjects[]?.name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[]? != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +### 5.1.9 Minimize access to create persistent volumes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to PersistentVolume objects in the cluster. + +### 5.1.10 Minimize access to the proxy sub-resource of nodes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the proxy sub-resource of node objects. + +### 5.1.11 Minimize access to the approval sub-resource of certificatesigningrequests objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the approval sub-resource of certificatesigningrequest objects. + +### 5.1.12 Minimize access to webhook configuration objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the validatingwebhookconfigurations or mutatingwebhookconfigurations objects + +### 5.1.13 Minimize access to the service account token creation (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the token sub-resource of serviceaccount objects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +### 5.2.7 Minimize the admission of root containers (Automated) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Automated) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md deleted file mode 100644 index 16614a7e520..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md +++ /dev/null @@ -1,3148 +0,0 @@ ---- -title: K3s Self-Assessment Guide - CIS Benchmark v1.7 - K8s v1.25 ---- - -This document is a companion to the [K3s Hardening Guide](../../../../pages-for-subheaders/k3s-hardening-guide.md), which provides prescriptive guidance on how to harden K3s clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 | - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.7. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -Each control in the CIS Kubernetes Benchmark was evaluated against a K3s cluster that was configured according to the accompanying hardening guide. - -Where control audits differ from the original CIS benchmark, the audit commands specific to K3s are provided for testing. - -These are the possible results for each control: - -- **Pass** - The K3s cluster passes the audit outlined in the benchmark. -- **Not Applicable** - The control is not applicable to K3s because of how it is designed to operate. The remediation section explains why. -- **Warn** - The control is manual in the CIS benchmark and it depends on the cluster's use-case or some other factor that must be determined by the cluster operator. These controls have been evaluated to ensure K3s doesn't prevent their implementation, but no further configuration or auditing of the cluster has been performed. - -This guide makes the assumption that K3s is running as a Systemd unit. Your installation may vary. Adjust the "audit" commands to fit your scenario. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Control Plane Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the -control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-apiserver.yaml - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-controller-manager.yaml - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /etc/kubernetes/manifests/kube-scheduler.yaml - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 /etc/kubernetes/manifests/etcd.yaml - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /etc/kubernetes/manifests/etcd.yaml - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 1.1.11 -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** Not Applicable - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/admin.conf - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 scheduler - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root scheduler - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 controllermanager - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/controller.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root controllermanager - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown -R root:root /etc/kubernetes/pki/ - -**Audit:** - -```bash -find /var/lib/rancher/k3s/server/tls | xargs stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) - - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - -**Audit:** - -```bash -stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) - - -**Result:** warn - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /etc/kubernetes/pki/*.key - -**Audit:** - -```bash -stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep containerd | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' is present OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 519 1 0 22:09 ? 00:00:00 /usr/bin/containerd root 801 1 0 22:09 ? 00:00:00 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 3864 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id d00174abbc275f6bb85c7f0be1d3154b9c91982a10b9dba6b5cb280f4d4c531d -address /run/k3s/containerd/containerd.sock root 4105 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7c2b546b4d2380bcb51278661f34cff94fad2ba06978e13f8f1b92dafcc89d43 -address /run/k3s/containerd/containerd.sock root 4206 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 68d8a55ff4663985be004608dbf78b0362f5522e18490c81d4c8dc9963de1556 -address /run/k3s/containerd/containerd.sock root 5374 1 0 22:31 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id ca0ae9e0b37dfd7b1ce05f72e1bc5a1be8f5cb08f2b4543081536de3bdbc925d -address /run/k3s/containerd/containerd.sock root 5443 1 0 22:31 ? 00:00:01 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3ea3c1cdbbd5adb8efd5c67a46aadd0fca9918dc0ad1f7cafe38b83171e3dc1b -address /run/k3s/containerd/containerd.sock root 7130 1 0 22:32 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4d838297d35a31003106ac5989c3547433985bb2964b47baad12cee6e375645e -address /run/k3s/containerd/containerd.sock root 7639 1 0 22:32 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 341cb9bcd8486aa2f1acb8e1ae51baebd630ac6ed266643266c34d677f61c7d0 -address /run/k3s/containerd/containerd.sock root 10308 1 0 23:17 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id c534fbee8e0d06fd9b29bf8fc70a138975c6b18db25f1faf2615677dfdb4199e -address /run/k3s/containerd/containerd.sock root 11370 1 0 23:18 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4ff4b8776dac7a35b83616d341dbe4d5a689ac7fb9b8eee8db5978e3968380ea -address /run/k3s/containerd/containerd.sock root 13736 13723 2 23:21 ? 00:00:10 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 16022 1 0 23:29 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9027256349086e458119478e5e00384b1b76fbf5e6dbee23699f596a88d9f2bc -address /run/k3s/containerd/containerd.sock root 16159 1 0 23:29 ? 00:00:00 /var/lib/rancher/k3s/data/630c40ff866a3db218a952ebd4fd2a5cfe1543a1a467e738cb46a2ad4012d6f1/bin/containerd-shim-runc-v2 -namespace k8s.io -id 929bf369fc5881654f4c1925624151ddb7cea51073267b8d213d966ba45406f3 -address /run/k3s/containerd/containerd.sock -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'AlwaysPullImages' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'ServiceAccount' -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-path' -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxage' -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxbackup' -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxsize' -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.23 Ensure that the --request-timeout argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameter as appropriate and if needed. -For example, --request-timeout=300s - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'request-timeout' -``` - -**Expected Result**: - -```console -'--request-timeout' is not present OR '--request-timeout' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-lookup' -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.25 Ensure that the --service-account-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-key-file' -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 1.2.29 -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'encryption-provider-config' -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 1.2.31 Ensure that encryption providers are appropriately configured (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit:** - -```bash -grep aescbc /path/to/encryption-config.json -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) - - -**Result:** warn - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'bind-address' -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true" -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.1 -``` - -**Expected Result**: - -```console -'cert-file' is present AND 'key-file' is present -``` - -**Returned Value**: - -```console -cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key -``` - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and set the below parameter. ---client-cert-auth="true" - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.2 -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -client-cert-auth: true -``` - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.3 -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.4 -``` - -**Expected Result**: - -```console -'cert-file' is present AND 'key-file' is present -``` - -**Returned Value**: - -```console -cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key -``` - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and set the below parameter. ---peer-client-cert-auth=true - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.5 -``` - -**Expected Result**: - -```console -'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -client-cert-auth: true -``` - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.6 -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit Script:** `check_for_k3s_etcd.sh` - -```bash -#!/bin/bash - -# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) -# before it checks the requirement -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - - -if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then - case $1 in - "1.1.11") - echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; - "1.2.29") - echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; - "2.1") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.2") - echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.3") - echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.4") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; - "2.5") - echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; - "2.6") - echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; - "2.7") - echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; - esac -else -# If another database is running, return whatever is required to pass the scan - case $1 in - "1.1.11") - echo "700";; - "1.2.29") - echo "--etcd-certfile AND --etcd-keyfile";; - "2.1") - echo "cert-file AND key-file";; - "2.2") - echo "--client-cert-auth=true";; - "2.3") - echo "false";; - "2.4") - echo "peer-cert-file AND peer-key-file";; - "2.5") - echo "--client-cert-auth=true";; - "2.6") - echo "--peer-auto-tls=false";; - "2.7") - echo "--trusted-ca-file";; - esac -fi - -``` - -**Audit Execution:** - -```bash -./check_for_k3s_etcd.sh 2.7 -``` - -**Expected Result**: - -```console -'trusted-ca-file' is present -``` - -**Returned Value**: - -```console -trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Manual) - - -**Result:** warn - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig -``` - -**Expected Result**: - -```console -'permissions' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is present OR '/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/agent/kubelet.kubeconfig -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit:** - -```bash -stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt -``` - -**Expected Result**: - -```console -'644' is equal to '644' OR '640' is present OR '600' is present OR '444' is present OR '440' is present OR '400' is present OR '000' is present -``` - -**Returned Value**: - -```console -644 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" | grep -v grep; else echo "--anonymous-auth=false"; fi' -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" | grep -v grep; else echo "--authorization-mode=Webhook"; fi' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" | grep -v grep; else echo "--client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt"; fi' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:42 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:42Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'protect-kernel-defaults' -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' -``` - -**Expected Result**: - -```console -'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -Feb 27 23:21:44 ip-172-31-31-124 k3s[13723]: time="2023-02-27T23:21:44Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-124 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c42d922-ed1e-4e15-8414-d399d179d897 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC containerd -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Manual) - - -**Result:** warn - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** warn - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) - - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md new file mode 100644 index 00000000000..903e42374f3 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md @@ -0,0 +1,515 @@ +--- +title: RKE 加固指南 +--- + + + + + +本文档提供了针对生产环境的 RKE 集群进行加固的具体指导，以便在使用 Rancher 部署之前进行配置。它概述了满足信息安全中心（Center for Information Security, CIS）Kubernetes benchmark controls 所需的配置和控制。 + +:::note +这份加固指南描述了如何确保你集群中的节点安全。我们建议你在安装 Kubernetes 之前遵循本指南。 +::: + +此加固指南适用于 RKE 集群，并与以下版本的 CIS Kubernetes Benchmark、Kubernetes 和 Rancher 相关联： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|------------------------------| +| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 至 v1.26 | + +:::note +- 在 Benchmark v1.24 及更高版本中，检查 id `4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automated)` 可能会失败，因为 `/etc/kubernetes/ssl/kube-ca.pem` 默认设置为 644。 +- 在 Benchmark v1.7 中，不再需要 `--protect-kernel-defaults` (`4.2.6`) 参数，并已被 CIS 删除。 +::: + +有关如何评估加固的 RKE 集群与官方 CIS benchmark 的更多细节，请参考特定 Kubernetes 和 CIS benchmark 版本的 RKE 自我评估指南。 + +## 主机级别要求 + +### 配置 Kernel 运行时参数 + +建议对群集中的所有节点类型使用以下 `sysctl` 配置。在 `/etc/sysctl.d/90-kubelet.conf` 中设置以下参数： + +```ini +vm.overcommit_memory=1 +vm.panic_on_oom=0 +kernel.panic=10 +kernel.panic_on_oops=1 +``` + +运行 `sysctl -p /etc/sysctl.d/90-kubelet.conf` 以启用设置。 + +### 配置 `etcd` 用户和组 + +在安装 RKE 之前，需要设置 **etcd** 服务的用户帐户和组。 + +#### 创建 `etcd` 用户和组 + +要创建 **etcd** 用户和组，请运行以下控制台命令。 +下面的命令示例中使用 `52034` 作为 **uid** 和 **gid** 。 +任何有效且未使用的 **uid** 或 **gid** 都可以代替 `52034`。 + +```bash +groupadd --gid 52034 etcd +useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd --shell /usr/sbin/nologin +``` + +在通过集群配置文件 `config.yml` 部署RKE时，请更新 `etcd` 用户的 `uid` 和 `gid`： + +```yaml +services: + etcd: + gid: 52034 + uid: 52034 +``` + +## Kubernetes 运行时要求 + +### 配置 `default` Service Account + +#### 设置 `automountServiceAccountToken` 为 `false` 用于 `default` service accounts + +Kubernetes 提供了一个 default service account，供集群工作负载使用，其中没有为 pod 分配特定的 service account。 +如果需要从 pod 访问 Kubernetes API，则应为该 pod 创建特定的 service account，并向该 service account 授予权限。 +应配置 default service account，使其不提供 service account 令牌，并且不应具有任何明确的权限分配。 + +对于标准 RKE 安装上的每个命名空间（包括 `default` 和 `kube-system`），`default` service account 必须包含以下值： + +```yaml +automountServiceAccountToken: false +``` + +将以下配置保存到名为 `account_update.yaml` 的文件中。 + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: default +automountServiceAccountToken: false +``` + +创建一个名为 `account_update.yaml` 的 bash 脚本文件。 +确保执行 `chmod +x account_update.sh` 命令，以赋予脚本执行权限。 + +```bash +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" +done +``` + +执行此脚本将 `account_update.yaml` 配置应用到所有命名空间中的 `default` service account。 + +### 配置网络策略 + +#### 确保所有命名空间都定义了网络策略 + +在同一个 Kubernetes 集群上运行不同的应用程序会带来风险，即某个受感染的应用程序可能会攻击相邻的应用程序。为确保容器只与其预期通信的容器进行通信，网络分段至关重要。网络策略规定了哪些 Pod 可以互相通信，以及与其他网络终端通信的方式。 + +网络策略是命名空间范围的。当在特定命名空间引入网络策略时，所有未被策略允许的流量将被拒绝。然而，如果在命名空间中没有网络策略，那么所有流量将被允许进入和离开该命名空间中的 Pod。要强制执行网络策略，必须启用容器网络接口（container network interface, CNI）插件。本指南使用 [Canal](https://github.com/projectcalico/canal) 来提供策略执行。有关 CNI 提供程序的其他信息可以在[这里](https://www.suse.com/c/rancher_blog/comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/)找到。 + +一旦在集群上启用了 CNI 提供程序，就可以应用默认的网络策略。下面提供了一个 **permissive** 的示例供参考。如果你希望允许匹配某个命名空间中所有 Pod 的所有入站和出站流量（即使添加了策略导致某些 Pod 被视为”隔离”），你可以创建一个明确允许该命名空间中所有流量的策略。请将以下配置保存为 `default-allow-all.yaml`。有关网络策略的其他[文档](https://kubernetes.io/docs/concepts/services-networking/network-policies/)可以在 Kubernetes 站点上找到。 + +:::caution +此网络策略只是一个示例，不建议用于生产用途。 +::: + +```yaml +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-allow-all +spec: + podSelector: {} + ingress: + - {} + egress: + - {} + policyTypes: + - Ingress + - Egress +``` + +创建一个名为 `apply_networkPolicy_to_all_ns.sh`的 Bash 脚本文件。 + +确保运行 `chmod +x apply_networkPolicy_to_all_ns.sh` 命令，以赋予脚本执行权限。 + +```bash +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + kubectl apply -f default-allow-all.yaml -n ${namespace} +done +``` + +执行此脚本以将 `default-allow-all.yaml` 配置和 **permissive** 的 `NetworkPolicy` 应用于所有命名空间。 + +## 已知限制 + +- 当注册自定义节点仅提供公共 IP 时，Rancher **exec shell** 和 **查看 pod 日志** 在加固设置中**不起作用**。 此功能需要在注册自定义节点时提供私有 IP。 +- 当根据 Rancher [提供](../../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md)的 Pod 安全策略 (Pod Security Policies, PSP) 将 `default_pod_security_policy_template_id:` 设置为 `restricted` 或 `restricted-noroot` 时，Rancher 会在 `default` service accounts 上创建 `RoleBindings` 和 `ClusterRoleBindings`。CIS 检查 5.1.5 要求除了默认角色之外，`default` service accounts 不应绑定其他角色或集群角色。此外，`default` service accounts 应配置为不提供服务账户令牌，也不具有任何明确的权限分配。 + +## 加固的 RKE `cluster.yml` 配置参考 + +参考的 `cluster.yml` 文件是由 RKE CLI 使用的，它提供了实现 RKE 加固安装所需的配置。 +RKE [文档](https://rancher.com/docs/rke/latest/en/installation/)提供了有关配置项的更多详细信息。这里参考的 `cluster.yml` 不包括必需的 `nodes` 指令，因为它取决于你的环境。在 RKE 中有关节点配置的文档可以在[这里](https://rancher.com/docs/rke/latest/en/config-options/nodes/)找到。 + +示例 `cluster.yml` 配置文件中包含了一个 Admission Configuration 策略，在 `services.kube-api.admission_configuration` 字段中指定。这个[示例](../../psa-restricted-exemptions.md)策略包含了命名空间的豁免规则，这对于在Rancher中正确运行导入的RKE集群非常必要，类似于Rancher预定义的 [`rancher-restricted`](../../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md) 策略。 + +如果你希望使用 RKE 的默认 `restricted` 策略，则将 `services.kube-api.admission_configuration` 字段留空，并将 `services.pod_security_configuration` 设置为 `restricted`。你可以在 [RKE 文档](https://rke.docs.rancher.com/config-options/services/pod-security-admission)中找到更多信息。 + + + + +:::note +如果你打算将一个 RKE 集群导入到 Rancher 中，请参考此[文档](../../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md)以了解如何配置 PSA 以豁免 Rancher 系统命名空间。 +::: + +```yaml +# 如果你打算在离线环境部署 Kubernetes， +# 请查阅文档以了解如何配置自定义的 RKE 镜像。 +nodes: [] +kubernetes_version: # 定义 RKE 版本 +services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + secrets_encryption_config: + enabled: true + audit_log: + enabled: true + event_rate_limit: + enabled: true + # 如果你在 `admission_configuration` 中设置了自定义策略， + # 请将 `pod_security_configuration` 字段留空。 + # 否则，将其设置为 `restricted` 以使用 RKE 预定义的受限策略， + # 并删除 `admission_configuration` 字段中的所有内容。 + # + # pod_security_configuration: restricted + # + admission_configuration: + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1 + kind: PodSecurityConfiguration + defaults: + enforce: "restricted" + enforce-version: "latest" + audit: "restricted" + audit-version: "latest" + warn: "restricted" + warn-version: "latest" + exemptions: + usernames: [] + runtimeClasses: [] + namespaces: [calico-apiserver, + calico-system, + cattle-alerting, + cattle-csp-adapter-system, + cattle-elemental-system, + cattle-epinio-system, + cattle-externalip-system, + cattle-fleet-local-system, + cattle-fleet-system, + cattle-gatekeeper-system, + cattle-global-data, + cattle-global-nt, + cattle-impersonation-system, + cattle-istio, + cattle-istio-system, + cattle-logging, + cattle-logging-system, + cattle-monitoring-system, + cattle-neuvector-system, + cattle-prometheus, + cattle-provisioning-capi-system, + cattle-resources-system, + cattle-sriov-system, + cattle-system, + cattle-ui-plugin-system, + cattle-windows-gmsa-system, + cert-manager, + cis-operator-system, + fleet-default, + ingress-nginx, + istio-system, + kube-node-lease, + kube-public, + kube-system, + longhorn-system, + rancher-alerting-drivers, + security-scan, + tigera-operator] + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + generate_serving_certificate: true +addons: | + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: default-allow-all + spec: + podSelector: {} + ingress: + - {} + egress: + - {} + policyTypes: + - Ingress + - Egress + --- + apiVersion: v1 + kind: ServiceAccount + metadata: + name: default + automountServiceAccountToken: false +``` + + + + +```yaml +# 如果你打算在离线环境部署 Kubernetes， +# 请查阅文档以了解如何配置自定义的 RKE 镜像。 +nodes: [] +kubernetes_version: # 定义 RKE 版本 +services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + secrets_encryption_config: + enabled: true + audit_log: + enabled: true + event_rate_limit: + enabled: true + pod_security_policy: true + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: true + generate_serving_certificate: true +addons: | + # Upstream Kubernetes restricted PSP policy + # https://github.com/kubernetes/website/blob/564baf15c102412522e9c8fc6ef2b5ff5b6e766c/content/en/examples/policy/restricted-psp.yaml + apiVersion: policy/v1beta1 + kind: PodSecurityPolicy + metadata: + name: restricted-noroot + spec: + privileged: false + # Required to prevent escalations to root. + allowPrivilegeEscalation: false + requiredDropCapabilities: + - ALL + # Allow core volume types. + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'downwardAPI' + # Assume that ephemeral CSI drivers & persistentVolumes set up by the cluster admin are safe to use. + - 'csi' + - 'persistentVolumeClaim' + - 'ephemeral' + hostNetwork: false + hostIPC: false + hostPID: false + runAsUser: + # Require the container to run without root privileges. + rule: 'MustRunAsNonRoot' + seLinux: + # This policy assumes the nodes are using AppArmor rather than SELinux. + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + # Forbid adding the root group. + - min: 1 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + # Forbid adding the root group. + - min: 1 + max: 65535 + readOnlyRootFilesystem: false + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: psp:restricted-noroot + rules: + - apiGroups: + - extensions + resourceNames: + - restricted-noroot + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: psp:restricted-noroot + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted-noroot + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: default-allow-all + spec: + podSelector: {} + ingress: + - {} + egress: + - {} + policyTypes: + - Ingress + - Egress + --- + apiVersion: v1 + kind: ServiceAccount + metadata: + name: default + automountServiceAccountToken: false +``` + + + + +## 加固后的 RKE 集群模板配置参考 + +参考的 RKE 集群模板提供了实现 Kubernetes 加固安装所需的最低配置。RKE 模板用于提供 Kubernetes 并定义 Rancher 设置。有关安装 RKE 及其模板详情的其他信息，请参考 Rancher [文档](../../../../getting-started/installation-and-upgrade/installation-and-upgrade.md) 。 + + + + +```yaml +# +# 集群配置 +# +default_pod_security_admission_configuration_template_name: rancher-restricted +enable_network_policy: true +local_cluster_auth_endpoint: + enabled: true +name: # 定义集群名称 + +# +# Rancher 配置 +# +rancher_kubernetes_engine_config: + addon_job_timeout: 45 + authentication: + strategy: x509|webhook + kubernetes_version: # 定义 RKE 版本 + services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + audit_log: + enabled: true + event_rate_limit: + enabled: true + pod_security_policy: false + secrets_encryption_config: + enabled: true + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + generate_serving_certificate: true + scheduler: + extra_args: + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + + + + +```yaml +# +# 集群配置 +# +default_pod_security_policy_template_id: restricted-noroot +enable_network_policy: true +local_cluster_auth_endpoint: + enabled: true +name: # 定义集群名称 + +# +# Rancher 配置 +# +rancher_kubernetes_engine_config: + addon_job_timeout: 45 + authentication: + strategy: x509|webhook + kubernetes_version: # 定义 RKE 版本 + services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + audit_log: + enabled: true + event_rate_limit: + enabled: true + pod_security_policy: true + secrets_encryption_config: + enabled: true + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + generate_serving_certificate: true + scheduler: + extra_args: + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + + + + +## 结论 + +如果你按照本指南操作，由 Rancher 提供的 RKE 自定义集群将配置为通过 CIS Kubernetes Benchmark 测试。你可以查看我们的 RKE 自我评估指南，了解我们是如何验证每个 benchmarks 的，并且你可以在你的集群上执行相同的操作。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md index 34bb63f89f0..7dfb1204916 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md @@ -1,30 +1,34 @@ --- -title: RKE Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.23 +title: RKE 自我评估指南 - CIS Benchmark v1.23 - K8s v1.23 --- -This document is a companion to the [RKE Hardening Guide](../../../../pages-for-subheaders/rke1-hardening-guide.md), which provides prescriptive guidance on how to harden RKE clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. + + + -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: +本文档是 [RKE 加固指南](rke1-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 -| Rancher Version | CIS Benchmark Version | Kubernetes Version | +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | |-----------------|-----------------------|--------------------| | Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 -This document is for Rancher operators, security teams, auditors and decision makers. +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.23 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 -## Testing Methodology +## 测试方法 -Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. +Rancher 和 RKE 通过 Docker 容器安装 Kubernetes 服务。配置是通过初始化时传递给容器的参数定义的，而不是通过配置文件。 -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 :::note -This guide only covers `automated` (previously called `scored`) tests. +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md deleted file mode 100644 index 25108b848b5..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md +++ /dev/null @@ -1,3084 +0,0 @@ ---- -title: RKE Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.24 ---- - -This document is a companion to the [RKE Hardening Guide](../../../../pages-for-subheaders/rke1-hardening-guide.md), which provides prescriptive guidance on how to harden RKE clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.24 | - -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Control Plane Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 permissions=600 -``` - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root -``` - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit:** - -```bash -stat -c %a /node/var/lib/etcd -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** Not Applicable - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chown -R root:root /etc/kubernetes/pki/ - -**Audit Script:** `check_files_owner_in_dir.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the owner is set to root:root for -# the given directory and all the files in it -# -# inputs: -# $1 = /full/path/to/directory -# -# outputs: -# true/false - -INPUT_DIR=$1 - -if [[ "${INPUT_DIR}" == "" ]]; then - echo "false" - exit -fi - -if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then - echo "false" - exit -fi - -statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) -while read -r statInfoLine; do - f=$(echo ${statInfoLine} | cut -d' ' -f1) - p=$(echo ${statInfoLine} | cut -d' ' -f2) - - if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then - if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "root:root" ]]; then - echo "false" - exit - fi - fi -done <<< "${statInfoLines}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - -**Audit Script:** `check_files_permissions.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_permissions.sh '/node/etc/kubernetes/ssl/!(*key).pem' -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /etc/kubernetes/ssl/*key.pem - -**Audit Script:** `check_files_permissions.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_permissions.sh '/node/etc/kubernetes/ssl/*key.pem' -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 1.2.31 Ensure that encryption providers are appropriately configured (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit Script:** `check_encryption_provider_config.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to check the encrption provider config is set to aesbc -# -# outputs: -# true/false - -# TODO: Figure out the file location from the kube-apiserver commandline args -ENCRYPTION_CONFIG_FILE="/node/etc/kubernetes/ssl/encryption.yaml" - -if [[ ! -f "${ENCRYPTION_CONFIG_FILE}" ]]; then - echo "false" - exit -fi - -for provider in "$@" -do - if grep "$provider" "${ENCRYPTION_CONFIG_FILE}"; then - echo "true" - exit - fi -done - -echo "false" -exit - -``` - -**Audit Execution:** - -```bash -./check_encryption_provider_config.sh aescbc -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -- aescbc: true -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 5056 5035 2 18:52 ? 00:00:05 kube-controller-manager --cloud-provider= --terminated-pod-gc-threshold=1000 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --node-monitor-grace-period=40s --allow-untagged-cloud=true --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --configure-cloud-routes=false --leader-elect=true --profiling=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --pod-eviction-timeout=5m0s --v=2 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --enable-hostpath-provisioner=false --use-service-account-credentials=true -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5056 5035 2 18:52 ? 00:00:05 kube-controller-manager --cloud-provider= --terminated-pod-gc-threshold=1000 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --node-monitor-grace-period=40s --allow-untagged-cloud=true --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --configure-cloud-routes=false --leader-elect=true --profiling=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --pod-eviction-timeout=5m0s --v=2 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --enable-hostpath-provisioner=false --use-service-account-credentials=true -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 5056 5035 2 18:52 ? 00:00:05 kube-controller-manager --cloud-provider= --terminated-pod-gc-threshold=1000 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --node-monitor-grace-period=40s --allow-untagged-cloud=true --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --configure-cloud-routes=false --leader-elect=true --profiling=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --pod-eviction-timeout=5m0s --v=2 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --enable-hostpath-provisioner=false --use-service-account-credentials=true -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 5056 5035 2 18:52 ? 00:00:05 kube-controller-manager --cloud-provider= --terminated-pod-gc-threshold=1000 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --node-monitor-grace-period=40s --allow-untagged-cloud=true --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --configure-cloud-routes=false --leader-elect=true --profiling=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --pod-eviction-timeout=5m0s --v=2 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --enable-hostpath-provisioner=false --use-service-account-credentials=true -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 5056 5035 2 18:52 ? 00:00:05 kube-controller-manager --cloud-provider= --terminated-pod-gc-threshold=1000 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --node-monitor-grace-period=40s --allow-untagged-cloud=true --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --configure-cloud-routes=false --leader-elect=true --profiling=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --pod-eviction-timeout=5m0s --v=2 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --enable-hostpath-provisioner=false --use-service-account-credentials=true -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -Cluster provisioned by RKE handles certificate rotation directly through RKE. - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 5056 5035 2 18:52 ? 00:00:05 kube-controller-manager --cloud-provider= --terminated-pod-gc-threshold=1000 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --node-monitor-grace-period=40s --allow-untagged-cloud=true --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --configure-cloud-routes=false --leader-elect=true --profiling=false --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --pod-eviction-timeout=5m0s --v=2 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --enable-hostpath-provisioner=false --use-service-account-credentials=true -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5209 5190 0 18:53 ? 00:00:01 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 5209 5190 0 18:53 ? 00:00:01 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--cert-file' is present AND '--key-file' is present -``` - -**Returned Value**: - -```console -root 4720 4700 3 18:52 ? 00:00:08 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --advertise-client-urls=https://172.31.26.105:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --listen-peer-urls=https://172.31.26.105:2380 --initial-cluster=etcd-ip-172-31-26-105=https://172.31.26.105:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --heartbeat-interval=500 --data-dir=/var/lib/rancher/etcd/ --listen-client-urls=https://172.31.26.105:2379 --name=etcd-ip-172-31-26-105 --initial-cluster-state=new --client-cert-auth=true --election-timeout=5000 --initial-advertise-peer-urls=https://172.31.26.105:2380 root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false root 18033 17938 2 18:57 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -root 4720 4700 3 18:52 ? 00:00:08 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --advertise-client-urls=https://172.31.26.105:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --listen-peer-urls=https://172.31.26.105:2380 --initial-cluster=etcd-ip-172-31-26-105=https://172.31.26.105:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --heartbeat-interval=500 --data-dir=/var/lib/rancher/etcd/ --listen-client-urls=https://172.31.26.105:2379 --name=etcd-ip-172-31-26-105 --initial-cluster-state=new --client-cert-auth=true --election-timeout=5000 --initial-advertise-peer-urls=https://172.31.26.105:2380 root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false root 18033 17938 4 18:57 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-26-105 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem ETCDCTL_ENDPOINTS=https://172.31.26.105:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/root -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-cert-file' is present AND '--peer-key-file' is present -``` - -**Returned Value**: - -```console -root 4720 4700 3 18:52 ? 00:00:08 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --advertise-client-urls=https://172.31.26.105:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --listen-peer-urls=https://172.31.26.105:2380 --initial-cluster=etcd-ip-172-31-26-105=https://172.31.26.105:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --heartbeat-interval=500 --data-dir=/var/lib/rancher/etcd/ --listen-client-urls=https://172.31.26.105:2379 --name=etcd-ip-172-31-26-105 --initial-cluster-state=new --client-cert-auth=true --election-timeout=5000 --initial-advertise-peer-urls=https://172.31.26.105:2380 root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false root 18033 17938 3 18:57 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -root 4720 4700 3 18:52 ? 00:00:08 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --advertise-client-urls=https://172.31.26.105:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --listen-peer-urls=https://172.31.26.105:2380 --initial-cluster=etcd-ip-172-31-26-105=https://172.31.26.105:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --heartbeat-interval=500 --data-dir=/var/lib/rancher/etcd/ --listen-client-urls=https://172.31.26.105:2379 --name=etcd-ip-172-31-26-105 --initial-cluster-state=new --client-cert-auth=true --election-timeout=5000 --initial-advertise-peer-urls=https://172.31.26.105:2380 root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false root 18033 17938 2 18:57 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-26-105 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem ETCDCTL_ENDPOINTS=https://172.31.26.105:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/root -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--trusted-ca-file' is present -``` - -**Returned Value**: - -```console -root 4720 4700 3 18:52 ? 00:00:08 /usr/local/bin/etcd --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --advertise-client-urls=https://172.31.26.105:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --listen-peer-urls=https://172.31.26.105:2380 --initial-cluster=etcd-ip-172-31-26-105=https://172.31.26.105:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-26-105-key.pem --heartbeat-interval=500 --data-dir=/var/lib/rancher/etcd/ --listen-client-urls=https://172.31.26.105:2379 --name=etcd-ip-172-31-26-105 --initial-cluster-state=new --client-cert-auth=true --election-timeout=5000 --initial-advertise-peer-urls=https://172.31.26.105:2380 root 4882 4861 15 18:52 ? 00:00:42 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false root 18033 17938 2 18:57 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-policy-file' is present -``` - -**Returned Value**: - -```console -root 4882 4861 15 18:52 ? 00:00:43 kube-apiserver --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --api-audiences=unknown --audit-log-maxsize=100 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-servers=https://172.31.26.105:2379 --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --anonymous-auth=false --secure-port=6443 --runtime-config=policy/v1beta1/podsecuritypolicy=true --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --service-node-port-range=30000-32767 --service-account-issuer=rke --service-account-lookup=true --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --authentication-token-webhook-cache-ttl=5s --advertise-address=172.31.26.105 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --admission-control-config-file=/etc/kubernetes/admission.yaml --audit-log-maxbackup=10 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --requestheader-username-headers=X-Remote-User --requestheader-group-headers=X-Remote-Group --authorization-mode=Node,RBAC --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --cloud-provider= --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --bind-address=0.0.0.0 --audit-log-maxage=30 --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --service-cluster-ip-range=10.43.0.0/16 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allow-privileged=true --profiling=false -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-node.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/kubernetes/ssl/kubecfg-kube-node.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit:** - -```bash -stat -c permissions=%a /node/etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit:** - -```bash -stat -c %U:%G /node/etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--streaming-connection-idle-timeout' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--event-qps' is equal to '0' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE handles certificate rotation directly through RKE. - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 5796 5377 2 18:53 ? 00:00:06 kubelet --anonymous-auth=false --cloud-provider= --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --root-dir=/var/lib/kubelet --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --fail-swap-on=false --hostname-override=ip-172-31-26-105 --authentication-token-webhook=true --cgroups-per-qos=True --cluster-dns=10.43.0.10 --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --address=0.0.0.0 --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-26-105.pem --streaming-connection-idle-timeout=30m --v=2 --protect-kernel-defaults=true --container-runtime=remote --event-qps=0 --feature-gates=RotateKubeletServerCertificate=true --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --cluster-domain=cluster.local --pod-infra-container-image=rancher/mirrored-pause:3.6 --read-only-port=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --node-ip=172.31.26.105 --resolv-conf=/etc/resolv.conf --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -**Audit Script:** `check_for_default_sa.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` - -**Audit Execution:** - -```bash -./check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): serviceaccounts is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "serviceaccounts" in API group "" at the cluster scope Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-fleet-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-impersonation-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cis-operator-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-node-lease" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-public" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "local" true -``` - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** warn - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) - - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -**Audit Script:** `check_for_network_policies.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in $(kubectl get namespaces --all-namespaces -o json | jq -r '.items[].metadata.name'); do - policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') - if [[ ${policy_count} -eq 0 ]]; then - echo "false" - exit - fi -done - -echo "true" - -``` - -**Audit Execution:** - -```bash -./check_for_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is present -``` - -**Returned Value**: - -```console -true -``` - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Automated) - - -**Result:** pass - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. - -**Audit Script:** `check_for_default_ns.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count=$(kubectl get all -n default -o json | jq .items[] | jq -r 'select((.metadata.name!="kubernetes"))' | jq .metadata.name | wc -l) -if [[ ${count} -gt 0 ]]; then - echo "false" - exit -fi - -echo "true" - - -``` - -**Audit Execution:** - -```bash -./check_for_default_ns.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): replicationcontrollers is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "replicationcontrollers" in API group "" in the namespace "default" Error from server (Forbidden): services is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "services" in API group "" in the namespace "default" Error from server (Forbidden): daemonsets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "daemonsets" in API group "apps" in the namespace "default" Error from server (Forbidden): deployments.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "deployments" in API group "apps" in the namespace "default" Error from server (Forbidden): replicasets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "replicasets" in API group "apps" in the namespace "default" Error from server (Forbidden): statefulsets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "statefulsets" in API group "apps" in the namespace "default" Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default" Error from server (Forbidden): cronjobs.batch is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "cronjobs" in API group "batch" in the namespace "default" Error from server (Forbidden): jobs.batch is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "jobs" in API group "batch" in the namespace "default" true -``` - diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md deleted file mode 100644 index 0cec041061f..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md +++ /dev/null @@ -1,3085 +0,0 @@ ---- -title: RKE Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.25 ---- - -This document is a companion to the [RKE Hardening Guide](../../../../pages-for-subheaders/rke1-hardening-guide.md), which provides prescriptive guidance on how to harden RKE clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.25 | - -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Control Plane Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 permissions=600 -``` - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root -``` - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit:** - -```bash -stat -c %a /node/var/lib/etcd -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** Not Applicable - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chown -R root:root /etc/kubernetes/pki/ - -**Audit Script:** `check_files_owner_in_dir.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the owner is set to root:root for -# the given directory and all the files in it -# -# inputs: -# $1 = /full/path/to/directory -# -# outputs: -# true/false - -INPUT_DIR=$1 - -if [[ "${INPUT_DIR}" == "" ]]; then - echo "false" - exit -fi - -if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then - echo "false" - exit -fi - -statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) -while read -r statInfoLine; do - f=$(echo ${statInfoLine} | cut -d' ' -f1) - p=$(echo ${statInfoLine} | cut -d' ' -f2) - - if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then - if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "root:root" ]]; then - echo "false" - exit - fi - fi -done <<< "${statInfoLines}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - -**Audit Script:** `check_files_permissions.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_permissions.sh '/node/etc/kubernetes/ssl/!(*key).pem' -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /etc/kubernetes/ssl/*key.pem - -**Audit Script:** `check_files_permissions.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_permissions.sh '/node/etc/kubernetes/ssl/*key.pem' -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.31 Ensure that encryption providers are appropriately configured (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit Script:** `check_encryption_provider_config.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to check the encrption provider config is set to aesbc -# -# outputs: -# true/false - -# TODO: Figure out the file location from the kube-apiserver commandline args -ENCRYPTION_CONFIG_FILE="/node/etc/kubernetes/ssl/encryption.yaml" - -if [[ ! -f "${ENCRYPTION_CONFIG_FILE}" ]]; then - echo "false" - exit -fi - -for provider in "$@" -do - if grep "$provider" "${ENCRYPTION_CONFIG_FILE}"; then - echo "true" - exit - fi -done - -echo "false" -exit - -``` - -**Audit Execution:** - -```bash -./check_encryption_provider_config.sh aescbc -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -- aescbc: true -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -Cluster provisioned by RKE handles certificate rotation directly through RKE. - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5671 5649 0 22:01 ? 00:00:01 kube-scheduler --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 5671 5649 0 22:01 ? 00:00:01 kube-scheduler --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--cert-file' is present AND '--key-file' is present -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 5 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 4 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-51 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem ETCDCTL_ENDPOINTS=https://172.31.31.51:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-cert-file' is present AND '--peer-key-file' is present -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 2 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 3 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-51 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem ETCDCTL_ENDPOINTS=https://172.31.31.51:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--trusted-ca-file' is present -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 2 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-policy-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:34 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-node.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/kubernetes/ssl/kubecfg-kube-node.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit:** - -```bash -stat -c permissions=%a /node/etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit:** - -```bash -stat -c %U:%G /node/etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--streaming-connection-idle-timeout' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--event-qps' is equal to '0' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE handles certificate rotation directly through RKE. - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -**Audit Script:** `check_for_default_sa.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` - -**Audit Execution:** - -```bash -./check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): serviceaccounts is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "serviceaccounts" in API group "" at the cluster scope Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-fleet-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-impersonation-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cis-operator-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-node-lease" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-public" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "local" true -``` - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** warn - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) - - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -**Audit Script:** `check_for_network_policies.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in $(kubectl get namespaces --all-namespaces -o json | jq -r '.items[].metadata.name'); do - policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') - if [[ ${policy_count} -eq 0 ]]; then - echo "false" - exit - fi -done - -echo "true" - -``` - -**Audit Execution:** - -```bash -./check_for_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is present -``` - -**Returned Value**: - -```console -true -``` - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Automated) - - -**Result:** pass - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. - -**Audit Script:** `check_for_default_ns.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count=$(kubectl get all -n default -o json | jq .items[] | jq -r 'select((.metadata.name!="kubernetes"))' | jq .metadata.name | wc -l) -if [[ ${count} -gt 0 ]]; then - echo "false" - exit -fi - -echo "true" - - -``` - -**Audit Execution:** - -```bash -./check_for_default_ns.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): replicationcontrollers is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "replicationcontrollers" in API group "" in the namespace "default" Error from server (Forbidden): services is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "services" in API group "" in the namespace "default" Error from server (Forbidden): daemonsets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "daemonsets" in API group "apps" in the namespace "default" Error from server (Forbidden): deployments.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "deployments" in API group "apps" in the namespace "default" Error from server (Forbidden): replicasets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "replicasets" in API group "apps" in the namespace "default" Error from server (Forbidden): statefulsets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "statefulsets" in API group "apps" in the namespace "default" Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default" Error from server (Forbidden): cronjobs.batch is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "cronjobs" in API group "batch" in the namespace "default" Error from server (Forbidden): jobs.batch is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "jobs" in API group "batch" in the namespace "default" true -``` - diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md new file mode 100644 index 00000000000..2246a13f6d0 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md @@ -0,0 +1,3046 @@ +--- +title: RKE 自我评估指南 - CIS Benchmark v1.24 - K8s v1.24 +--- + + + + + +本文档是 [RKE 加固指南](rke1-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | + +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.24 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +Rancher 和 RKE 通过 Docker 容器安装 Kubernetes 服务。配置是通过初始化时传递给容器的参数定义的，而不是通过配置文件。 + +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. +All configuration is passed in as arguments at container run time. + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. +All configuration is passed in as arguments at container run time. + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=644 +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root root:root +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c %a /node/var/lib/etcd +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** Not Applicable + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit Script:** `check_files_owner_in_dir.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to ensure the owner is set to root:root for +# the given directory and all the files in it +# +# inputs: +# $1 = /full/path/to/directory +# +# outputs: +# true/false + +INPUT_DIR=$1 + +if [[ "${INPUT_DIR}" == "" ]]; then + echo "false" + exit +fi + +if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then + echo "false" + exit +fi + +statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) +while read -r statInfoLine; do + f=$(echo ${statInfoLine} | cut -d' ' -f1) + p=$(echo ${statInfoLine} | cut -d' ' -f2) + + if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then + if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "root:root" ]]; then + echo "false" + exit + fi + fi +done <<< "${statInfoLines}" + + +echo "true" +exit + +``` + +**Audit Execution:** + +```bash +./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' -exec chmod -R 600 {} + + +**Audit:** + +```bash +find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' | xargs stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=644 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/ssl/*key.pem + +**Audit Script:** `check_files_permissions.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to ensure the file permissions are set to 644 or +# more restrictive for all files in a given directory or a wildcard +# selection of files +# +# inputs: +# $1 = /full/path/to/directory or /path/to/fileswithpattern +# ex: !(*key).pem +# +# $2 (optional) = permission (ex: 600) +# +# outputs: +# true/false + +# Turn on "extended glob" for use of '!' in wildcard +shopt -s extglob + +# Turn off history to avoid surprises when using '!' +set -H + +USER_INPUT=$1 + +if [[ "${USER_INPUT}" == "" ]]; then + echo "false" + exit +fi + + +if [[ -d ${USER_INPUT} ]]; then + PATTERN="${USER_INPUT}/*" +else + PATTERN="${USER_INPUT}" +fi + +PERMISSION="" +if [[ "$2" != "" ]]; then + PERMISSION=$2 +fi + +FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) + +while read -r fileInfo; do + p=$(echo ${fileInfo} | cut -d' ' -f2) + + if [[ "${PERMISSION}" != "" ]]; then + if [[ "$p" != "${PERMISSION}" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then + echo "false" + exit + fi + fi +done <<< "${FILES_PERMISSIONS}" + + +echo "true" +exit + +``` + +**Audit Execution:** + +```bash +./check_files_permissions.sh '/node/etc/kubernetes/ssl/*key.pem' +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --kubelet-https parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-https' is present OR '--kubelet-https' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.18 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxsize' is greater or equal to 100 +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.31 Ensure that encryption providers are appropriately configured (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit Script:** `check_encryption_provider_config.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to check the encrption provider config is set to aesbc +# +# outputs: +# true/false + +# TODO: Figure out the file location from the kube-apiserver commandline args +ENCRYPTION_CONFIG_FILE="/node/etc/kubernetes/ssl/encryption.yaml" + +if [[ ! -f "${ENCRYPTION_CONFIG_FILE}" ]]; then + echo "false" + exit +fi + +for provider in "$@" +do + if grep "$provider" "${ENCRYPTION_CONFIG_FILE}"; then + echo "true" + exit + fi +done + +echo "false" +exit + +``` + +**Audit Execution:** + +```bash +./check_encryption_provider_config.sh aescbc +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +- aescbc: true +``` + +### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +Cluster provisioned by RKE handles certificate rotation directly through RKE. + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 3859 3838 0 Sep11 ? 00:03:44 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 3859 3838 0 Sep11 ? 00:03:44 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--cert-file' is present AND '--key-file' is present +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:08 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 5 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:08 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 4 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-7-100 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-cert-file' is present AND '--peer-key-file' is present +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 2 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 2 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-7-100 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--trusted-ca-file' is present +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 2 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:10 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. +All configuration is passed in as arguments at container run time. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. +All configuration is passed in as arguments at container run time. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit:** + +```bash +stat -c permissions=%a /node/etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit:** + +```bash +stat -c %U:%G /node/etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/kubelet/config.yaml + +Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml + +Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--streaming-connection-idle-timeout' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--protect-kernel-defaults' is equal to 'true' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors + +### 4.2.9 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--event-qps' is equal to '0' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +Clusters provisioned by RKE handles certificate rotation directly through RKE. + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Automated) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[]?.kind=="ServiceAccount" and .subjects[]?.name=="default") or (.subjects[]?.kind=="Group" and .subjects[]?.name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[]? != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=1 +``` + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=1 +``` + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=1 +``` + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=1 +``` + +### 5.2.7 Minimize the admission of root containers (Manual) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +**Audit Script:** `check_for_network_policies.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +for namespace in $(kubectl get namespaces --all-namespaces -o json | jq -r '.items[].metadata.name'); do + policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') + if [[ ${policy_count} -eq 0 ]]; then + echo "false" + exit + fi +done + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_network_policies.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Automated) + + +**Result:** pass + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + +**Audit Script:** `check_for_default_ns.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count=$(kubectl get all -n default -o json | jq .items[] | jq -r 'select((.metadata.name!="kubernetes"))' | jq .metadata.name | wc -l) +if [[ ${count} -gt 0 ]]; then + echo "false" + exit +fi + +echo "true" + + +``` + +**Audit Execution:** + +```bash +./check_for_default_ns.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md new file mode 100644 index 00000000000..5e5a6b1eed2 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md @@ -0,0 +1,2862 @@ +--- +title: RKE 自我评估指南 - CIS Benchmark v1.7 - K8s v1.25/v1.26/v1.27 +--- + + + + + +本文档是 [RKE 加固指南](rke1-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25/v1.26/v1.27 | + +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.7 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +Rancher 和 RKE 通过 Docker 容器安装 Kubernetes 服务。配置是通过初始化时传递给容器的参数定义的，而不是通过配置文件。 + +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-apiserver.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-controller-manager.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-scheduler.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /etc/kubernetes/manifests/etcd.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /etc/kubernetes/manifests/etcd.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +**Audit:** + +```bash +ps -ef | grep kubelet | grep -- --cni-conf-dir | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +**Audit:** + +```bash +ps -ef | grep kubelet | grep -- --cni-conf-dir | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c %a /node/var/lib/etcd +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +**Audit:** + +```bash +stat -c %U:%G /node/var/lib/etcd +``` + +**Expected Result**: + +```console +'etcd:etcd' is present +``` + +**Returned Value**: + +```console +etcd:etcd +``` + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/admin.conf +Not Applicable - Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf +Not Applicable - Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 scheduler +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root scheduler +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 controllermanager +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root controllermanager +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit Script:** `check_files_owner_in_dir.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to ensure the owner is set to root:root for +# the given directory and all the files in it +# +# inputs: +# $1 = /full/path/to/directory +# +# outputs: +# true/false + +INPUT_DIR=$1 + +if [[ "${INPUT_DIR}" == "" ]]; then + echo "false" + exit +fi + +if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then + echo "false" + exit +fi + +statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) +while read -r statInfoLine; do + f=$(echo ${statInfoLine} | cut -d' ' -f1) + p=$(echo ${statInfoLine} | cut -d' ' -f2) + + if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then + if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "root:root" ]]; then + echo "false" + exit + fi + fi +done <<< "${statInfoLines}" + + +echo "true" +exit + +``` + +**Audit Execution:** + +```bash +./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' -exec chmod -R 600 {} + + +**Audit:** + +```bash +find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' | xargs stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=644 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +find /node/etc/kubernetes/ssl/ -name '*key.pem' -exec chmod -R 600 {} + + +**Audit:** + +```bash +find /node/etc/kubernetes/ssl/ -name '*key.pem' | xargs stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= +When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. + +### 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.7 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.9 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.12 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.13 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.14 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.15 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.16 Ensure that the --secure-port argument is not set to 0 - NoteThis recommendation is obsolete and will be deleted per the consensus process (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.17 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.18 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.19 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.20 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.21 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxsize' is greater or equal to 100 +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.22 Ensure that the --request-timeout argument is set as appropriate (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, --request-timeout=300s + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +### 1.2.23 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.24 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.25 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.26 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.27 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.28 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.29 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.30 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit:** + +```bash +ENCRYPTION_PROVIDER_CONFIG=$(ps -ef | grep kube-apiserver | grep -- --encryption-provider-config | sed 's%.*encryption-provider-config[= ]\([^ ]*\).*%\1%') if test -e $ENCRYPTION_PROVIDER_CONFIG; then grep -A1 'providers:' $ENCRYPTION_PROVIDER_CONFIG | tail -n1 | grep -o "[A-Za-z]*" | sed 's/^/provider=/'; fi +``` + +**Expected Result**: + +```console +'provider' is present +``` + +### 1.2.31 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true +Cluster provisioned by RKE handles certificate rotation directly through RKE. + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 4339 4318 0 Sep11 ? 00:03:28 kube-scheduler --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --profiling=false --v=2 --leader-elect=true +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 4339 4318 0 Sep11 ? 00:03:28 kube-scheduler --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --profiling=false --v=2 --leader-elect=true +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--cert-file' is present AND '--key-file' is present +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 2 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 1 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-4-224 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-cert-file' is present AND '--peer-key-file' is present +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 2 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 1 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-4-224 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--trusted-ca-file' is present +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 1 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +### 3.1.2 Service account token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of service account tokens. + +### 3.1.3 Bootstrap token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of bootstrap tokens. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable - Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. +All configuration is passed in as arguments at container run time. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable - Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. + All configuration is passed in as arguments at container run time. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit:** + +```bash +stat -c permissions=%a /node/etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit:** + +```bash +stat -c %U:%G /node/etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/kubelet/config.yaml +Not Applicable - Clusters provisioned by RKE do not require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml +Not Applicable - Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--streaming-connection-idle-timeout' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.7 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable - Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors + +### 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--event-qps' is greater or equal to 0 OR '--event-qps' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.10 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable - Clusters provisioned by RKE handles certificate rotation directly through RKE. + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +### 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.13 Ensure that a limit is set on pod PIDs (Manual) + + +**Result:** warn + +**Remediation:** +Decide on an appropriate level for this parameter and set it, +either via the --pod-max-pids command line parameter or the PodPidsLimit configuration file setting. + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--pod-max-pids' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[]?.kind=="ServiceAccount" and .subjects[]?.name=="default") or (.subjects[]?.kind=="Group" and .subjects[]?.name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[]? != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +### 5.1.9 Minimize access to create persistent volumes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to PersistentVolume objects in the cluster. + +### 5.1.10 Minimize access to the proxy sub-resource of nodes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the proxy sub-resource of node objects. + +### 5.1.11 Minimize access to the approval sub-resource of certificatesigningrequests objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the approval sub-resource of certificatesigningrequest objects. + +### 5.1.12 Minimize access to webhook configuration objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the validatingwebhookconfigurations or mutatingwebhookconfigurations objects + +### 5.1.13 Minimize access to the service account token creation (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the token sub-resource of serviceaccount objects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +### 5.2.7 Minimize the admission of root containers (Manual) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md deleted file mode 100644 index 084568cc402..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md +++ /dev/null @@ -1,3085 +0,0 @@ ---- -title: RKE Self-Assessment Guide - CIS Benchmark v1.7 - K8s v1.25 ---- - -This document is a companion to the [RKE Hardening Guide](../../../../pages-for-subheaders/rke1-hardening-guide.md), which provides prescriptive guidance on how to harden RKE clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 | - -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.7. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Control Plane Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. -All configuration is passed in as arguments at container run time. - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. -All configuration is passed in as arguments at container run time. - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 permissions=600 -``` - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root -``` - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit:** - -```bash -stat -c %a /node/var/lib/etcd -``` - -**Expected Result**: - -```console -'700' is equal to '700' -``` - -**Returned Value**: - -```console -700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** Not Applicable - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. -All configuration is passed in as arguments at container run time. - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. -All configuration is passed in as arguments at container run time. - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chown -R root:root /etc/kubernetes/pki/ - -**Audit Script:** `check_files_owner_in_dir.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the owner is set to root:root for -# the given directory and all the files in it -# -# inputs: -# $1 = /full/path/to/directory -# -# outputs: -# true/false - -INPUT_DIR=$1 - -if [[ "${INPUT_DIR}" == "" ]]; then - echo "false" - exit -fi - -if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then - echo "false" - exit -fi - -statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) -while read -r statInfoLine; do - f=$(echo ${statInfoLine} | cut -d' ' -f1) - p=$(echo ${statInfoLine} | cut -d' ' -f2) - - if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then - if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "root:root" ]]; then - echo "false" - exit - fi - fi -done <<< "${statInfoLines}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the master node. -For example, -chmod -R 644 /etc/kubernetes/pki/*.crt - -**Audit Script:** `check_files_permissions.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_permissions.sh '/node/etc/kubernetes/ssl/!(*key).pem' -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /etc/kubernetes/ssl/*key.pem - -**Audit Script:** `check_files_permissions.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to ensure the file permissions are set to 644 or -# more restrictive for all files in a given directory or a wildcard -# selection of files -# -# inputs: -# $1 = /full/path/to/directory or /path/to/fileswithpattern -# ex: !(*key).pem -# -# $2 (optional) = permission (ex: 600) -# -# outputs: -# true/false - -# Turn on "extended glob" for use of '!' in wildcard -shopt -s extglob - -# Turn off history to avoid surprises when using '!' -set -H - -USER_INPUT=$1 - -if [[ "${USER_INPUT}" == "" ]]; then - echo "false" - exit -fi - - -if [[ -d ${USER_INPUT} ]]; then - PATTERN="${USER_INPUT}/*" -else - PATTERN="${USER_INPUT}" -fi - -PERMISSION="" -if [[ "$2" != "" ]]; then - PERMISSION=$2 -fi - -FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) - -while read -r fileInfo; do - p=$(echo ${fileInfo} | cut -d' ' -f2) - - if [[ "${PERMISSION}" != "" ]]; then - if [[ "$p" != "${PERMISSION}" ]]; then - echo "false" - exit - fi - else - if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then - echo "false" - exit - fi - fi -done <<< "${FILES_PERMISSIONS}" - - -echo "true" -exit - -``` - -**Audit Execution:** - -```bash -./check_files_permissions.sh '/node/etc/kubernetes/ssl/*key.pem' -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -true -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 1.2.31 Ensure that encryption providers are appropriately configured (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit Script:** `check_encryption_provider_config.sh` - -```bash -#!/usr/bin/env bash - -# This script is used to check the encrption provider config is set to aesbc -# -# outputs: -# true/false - -# TODO: Figure out the file location from the kube-apiserver commandline args -ENCRYPTION_CONFIG_FILE="/node/etc/kubernetes/ssl/encryption.yaml" - -if [[ ! -f "${ENCRYPTION_CONFIG_FILE}" ]]; then - echo "false" - exit -fi - -for provider in "$@" -do - if grep "$provider" "${ENCRYPTION_CONFIG_FILE}"; then - echo "true" - exit - fi -done - -echo "false" -exit - -``` - -**Audit Execution:** - -```bash -./check_encryption_provider_config.sh aescbc -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -- aescbc: true -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -Cluster provisioned by RKE handles certificate rotation directly through RKE. - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 5506 5484 2 22:01 ? 00:00:05 kube-controller-manager --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --v=2 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --allow-untagged-cloud=true --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --cluster-cidr=10.42.0.0/16 --node-monitor-grace-period=40s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --service-cluster-ip-range=10.43.0.0/16 --profiling=false --configure-cloud-routes=false --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --use-service-account-credentials=true -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 5671 5649 0 22:01 ? 00:00:01 kube-scheduler --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is present OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 5671 5649 0 22:01 ? 00:00:01 kube-scheduler --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--cert-file' is present AND '--key-file' is present -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 5 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 4 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-51 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem ETCDCTL_ENDPOINTS=https://172.31.31.51:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-cert-file' is present AND '--peer-key-file' is present -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 2 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--peer-client-cert-auth' is equal to 'true' -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 3 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-51 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem ETCDCTL_ENDPOINTS=https://172.31.31.51:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'--trusted-ca-file' is present -``` - -**Returned Value**: - -```console -etcd 5188 5167 3 22:01 ? 00:00:08 /usr/local/bin/etcd --client-cert-auth=true --data-dir=/var/lib/rancher/etcd/ --initial-advertise-peer-urls=https://172.31.31.51:2380 --listen-peer-urls=https://172.31.31.51:2380 --initial-cluster=etcd-ip-172-31-31-51=https://172.31.31.51:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --peer-client-cert-auth=true --listen-client-urls=https://172.31.31.51:2379 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51.pem --advertise-client-urls=https://172.31.31.51:2379 --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-31-51 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-31-51-key.pem --election-timeout=5000 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --heartbeat-interval=500 root 5354 5332 14 22:01 ? 00:00:33 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem root 19036 18926 2 22:05 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.23-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-policy-file' is present -``` - -**Returned Value**: - -```console -root 5354 5332 14 22:01 ? 00:00:34 kube-apiserver --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --runtime-config=authorization.k8s.io/v1beta1=true --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxsize=100 --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-group-headers=X-Remote-Group --storage-backend=etcd3 --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --authentication-token-webhook-cache-ttl=5s --etcd-prefix=/registry --service-node-port-range=30000-32767 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --profiling=false --audit-log-format=json --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --allow-privileged=true --requestheader-username-headers=X-Remote-User --anonymous-auth=false --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --api-audiences=unknown --etcd-servers=https://172.31.31.51:2379 --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --advertise-address=172.31.31.51 --audit-log-maxage=30 --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --requestheader-extra-headers-prefix=X-Remote-Extra- --bind-address=0.0.0.0 --service-account-lookup=true --authorization-mode=Node,RBAC --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-allowed-names=kube-apiserver-proxy-client --service-account-issuer=rke --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --secure-port=6443 --audit-log-maxbackup=10 --audit-policy-file=/etc/kubernetes/audit-policy.yaml --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --service-cluster-ip-range=10.43.0.0/16 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. -All configuration is passed in as arguments at container run time. - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present OR '/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml' is not present -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /etc/kubernetes/ssl/kubecfg-kube-node.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/kubernetes/ssl/kubecfg-kube-node.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit:** - -```bash -stat -c permissions=%a /node/etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit:** - -```bash -stat -c %U:%G /node/etc/kubernetes/ssl/kube-ca.pem -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /var/lib/kubelet/config.yaml - -Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. -All configuration is passed in as arguments at container run time. - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--streaming-connection-idle-timeout' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--event-qps' is equal to '0' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -Clusters provisioned by RKE handles certificate rotation directly through RKE. - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' -``` - -**Expected Result**: - -```console -'--tls-cipher-suites' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 6239 5834 2 22:02 ? 00:00:04 kubelet --authorization-mode=Webhook --v=2 --root-dir=/var/lib/kubelet --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51-key.pem --cgroups-per-qos=True --streaming-connection-idle-timeout=30m --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-31-51.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --address=0.0.0.0 --cluster-domain=cluster.local --fail-swap-on=false --make-iptables-util-chains=true --volume-plugin-dir=/var/lib/kubelet/volumeplugins --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --pod-infra-container-image=rancher/mirrored-pause:3.6 --node-ip=172.31.31.51 --resolv-conf=/etc/resolv.conf --event-qps=0 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --protect-kernel-defaults=true --cluster-dns=10.43.0.10 --container-runtime=remote --authentication-token-webhook=true --anonymous-auth=false --feature-gates=RotateKubeletServerCertificate=true --cloud-provider= --read-only-port=0 --hostname-override=ip-172-31-31-51 --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -**Audit Script:** `check_for_default_sa.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` - -**Audit Execution:** - -```bash -./check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): serviceaccounts is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "serviceaccounts" in API group "" at the cluster scope Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-fleet-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-impersonation-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cis-operator-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-node-lease" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-public" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "local" true -``` - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** warn - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) - - -**Result:** warn - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -**Audit Script:** `check_for_network_policies.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in $(kubectl get namespaces --all-namespaces -o json | jq -r '.items[].metadata.name'); do - policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') - if [[ ${policy_count} -eq 0 ]]; then - echo "false" - exit - fi -done - -echo "true" - -``` - -**Audit Execution:** - -```bash -./check_for_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is present -``` - -**Returned Value**: - -```console -true -``` - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Automated) - - -**Result:** pass - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. - -**Audit Script:** `check_for_default_ns.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count=$(kubectl get all -n default -o json | jq .items[] | jq -r 'select((.metadata.name!="kubernetes"))' | jq .metadata.name | wc -l) -if [[ ${count} -gt 0 ]]; then - echo "false" - exit -fi - -echo "true" - - -``` - -**Audit Execution:** - -```bash -./check_for_default_ns.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): replicationcontrollers is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "replicationcontrollers" in API group "" in the namespace "default" Error from server (Forbidden): services is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "services" in API group "" in the namespace "default" Error from server (Forbidden): daemonsets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "daemonsets" in API group "apps" in the namespace "default" Error from server (Forbidden): deployments.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "deployments" in API group "apps" in the namespace "default" Error from server (Forbidden): replicasets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "replicasets" in API group "apps" in the namespace "default" Error from server (Forbidden): statefulsets.apps is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "statefulsets" in API group "apps" in the namespace "default" Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default" Error from server (Forbidden): cronjobs.batch is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "cronjobs" in API group "batch" in the namespace "default" Error from server (Forbidden): jobs.batch is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "jobs" in API group "batch" in the namespace "default" true -``` - diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md new file mode 100644 index 00000000000..431d87bb1ed --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md @@ -0,0 +1,273 @@ +--- +title: RKE2 加固指南 +--- + + + + + +本文档提供了针对生产环境的 RKE2 集群进行加固的具体指导，以便在使用 Rancher 部署之前进行配置。它概述了满足信息安全中心（Center for Information Security, CIS）Kubernetes benchmark controls 制所需的配置和控制。 + +:::note +这份加固指南描述了如何确保你集群中的节点安全。我们建议你在安装 Kubernetes 之前遵循本指南。 +::: + +此加固指南适用于 RKE2 集群，并与以下版本的 CIS Kubernetes Benchmark、Kubernetes 和 Rancher 相关联： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|------------------------------| +| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 至 v1.26 | + +:::note +- 在 Benchmark v1.24 及更高版本中，由于新的文件权限要求（600 而不是 644），一些检查 ID 可能会失败。受影响的检查 ID 包括：`1.1.1`, `1.1.3`, `1.1.5`, `1.1.7`, `1.1.13`, `1.1.15`, `1.1.17`, `4.1.3`, `4.1.5` 和 `4.1.9`。 +- 在 Benchmark v1.7 中，不再需要 `--protect-kernel-defaults` (4.2.6) 参数，并已被 CIS 删除。 +::: + +有关如何评估加固的 RKE2 集群与官方 CIS benchmark 的更多细节，请参考特定 Kubernetes 和 CIS benchmark 版本的 RKE2 自我评估指南。 + +RKE2 在不需要修改的情况下通过了许多 Kubernetes CIS controls，因为它默认应用了几个安全缓解措施。然而，有一些值得注意的例外情况，需要手动干预才能完全符合 CIS Benchmark 要求： + +1. RKE2 不会修改主机操作系统。因此，作为运维人员，你必须进行一些主机级别的修改。 +2. 某些 CIS controls 对于网络策略和 Pod 安全标准（或 RKE2 v1.25 之前的 Pod 安全策略 (PSP)）将限制集群的功能。你必须选择让 RKE2 为你配置这些功能。为了确保满足这些要求，可以启动 RKE2 并设置 profile 标志为 `cis-1.23`（适用于 v1.25 及更新版本）或 `cis-1.6`（适用于 v1.24 及更早版本）。 + +## 主机级别要求 + +主机级要求有两个方面：内核参数和 etcd 进程/目录配置。这些在本节中进行了概述。 + +### 确保 `protect-kernel-defaults` 已经设置 + + + + +自 CIS benchmark 1.7 开始，不再需要`protect-kernel-defaults`。 + + + + +这是一个 kubelet 标志，如果所需的内核参数未设置或设置为与 kubelet 的默认值不同的值，将导致 kubelet 退出。 + +可以在 Rancher 的集群配置中设置 `protect-kernel-defaults` 标志。 + +```yaml +spec: + rkeConfig: + machineSelectorConfig: + - config: + protect-kernel-defaults: true +``` + + + + +### 设置内核参数 + +建议为集群中的所有节点类型设置以下 `sysctl` 配置。在 `/etc/sysctl.d/90-kubelet.conf` 中设置以下参数： + +```ini +vm.panic_on_oom=0 +vm.overcommit_memory=1 +kernel.panic=10 +kernel.panic_on_oops=1 +``` + +运行 `sudo sysctl -p /etc/sysctl.d/90-kubelet.conf` 以启用设置。 + +### 确保 etcd 配置正确 + +CIS Benchmark 要求 etcd 数据目录由 `etcd` 用户和组拥有。这意味着 etcd 进程必须以主机级别的 `etcd` 用户身份运行。为了实现这一点，在使用有效的 `cis-1.xx` 配置文件启动 RKE2 时，RKE2 会采取以下几个步骤： + +1. 检查主机上是否存在 `etcd` 用户和组。如果不存在，则显示错误并退出。 +2. 使用 `etcd` 作为用户和组所有者创建 etcd 的数据目录。 +3. 通过适当设置 etcd 静态 Pod 的 `SecurityContext`，确保 etcd 进程以 `etcd` 用户和组的身份运行。 + +为满足上述要求，你必须执行以下操作： + +#### 创建 etcd 用户 + +在某些 Linux 发行版中，`useradd` 命令不会创建组。下面包含了 `-U` 标志来解决这个问题。这个标志告诉 `useradd` 创建一个与用户同名的组。 + +```bash +sudo useradd -r -c "etcd user" -s /sbin/nologin -M etcd -U +``` + +## Kubernetes 运行时要求 + +通过 CIS Benchmark 测试的运行时要求主要集中在 Pod 安全、网络策略和内核参数上。当使用有效的 `cis-1.xx` 配置文件时，大部分都会被 RKE2 自动处理，但需要一些额外的运维人员干预是。本节概述了这些内容。 + +### Pod 安全 + +RKE2 始终以一定程度的 Pod 安全性运行。 + + + + +在 v1.25 及更高版本中，[Pod 安全准入(PSAs)](https://kubernetes.io/docs/concepts/security/pod-security-admission/)用于保证 pod 安全。 + +以下是确保加固 RKE2 通过 Rancher 中提供的 CIS v1.23 加固配置文件 `rke2-cis-1.7-hardened` 所需的最低配置。 + +```yaml +spec: + defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted + rkeConfig: + machineSelectorConfig: + - config: + profile: cis-1.23 +``` + +当同时设置了 `defaultPodSecurityAdmissionConfigurationTemplateName` 和 `profile` 标志时，Rancher 和 RKE2 会执行以下操作： + +1. 检查是否已满足主机级要求。如果未满足，RKE2 将以致命错误退出，并描述未满足的要求。 +2. 应用允许群集传递关联 controls 的网络策略。 +3. 配置 Pod 安全准入控制器（PSA）使用 PSA 配置模板 `rancher-restricted`，以在所有命名空间中强制执行受限模式，除了模板豁免列表中的命名空间。 + 这些命名空间被豁免，以允许系统 Pod 在没有限制的情况下运行，这是集群正常运行所必需的。 + +:::note +如果你打算将一个 RKE 集群导入到 Rancher 中，请参考[文档](../../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md)了解如何配置 PSA 以豁免 Rancher system 命名空间。 +::: + + + + + +在 Kubernetes v1.24 及更早版本中，`PodSecurityPolicy` 准入控制器始终是启用的。 + +以下是确保 RKE2 加固以通过 Rancher 中提供的 CIS v1.23 加固配置文件 `rke2-cis-1.23-hardened` 所需的最低配置。 + +:::note +在下面的示例中，配置文件设置为`cis-1.6`，这是在上游 RKE2 中定义的值，但集群实际上配置为传递 CIS v1.23 加固配置文件 +::: + +```yaml +spec: + defaultPodSecurityPolicyTemplateName: restricted-noroot + rkeConfig: + machineSelectorConfig: + - config: + profile: cis-1.6 +``` + + +当同时设置了 `defaultPodSecurityPolicyTemplateName` 和 `profile` 标志时，Rancher 和 RKE2 会执行以下操作： + +1. 检查是否已满足主机级要求。如果未满足，RKE2 将以致命错误退出，并描述未满足的要求。 +2. 应用网络策略，以确保集群通过相关的 controls 要求。 +3. 配置运行时 Pod 安全策略，以确保集群通过相关的 controls 要求。 + + + + +:::note +Kubernetes control plane 组件以及关键的附加组件，如 CNI、DNS 和 Ingress，都作为 `kube-system` 命名空间中的 Pod 运行。因此，这个命名空间的限制政策较少，从而使这些组件能够正常运行。 +::: + +### 网络策略 + +当使用有效的 `cis-1.xx` 配置文件运行时，RKE2 将设置适当的 `NetworkPolicies`，以满足 Kubernetes 内置命名空间的 CIS Benchmark 要求。这些命名空间包括：`kube-system`、`kube-public`、`kube-node-lease` 和 `default`。 + +所使用的 `NetworkPolicy` 仅允许同一命名空间内的 Pod 相互通信。值得注意的例外是允许 DNS 请求进行解析。 + +:::note +运维人员必须像管理其他命名空间一样管理额外创建的命名空间的网络策略。 +::: + +### 配置 `default` service account + +**将 `default` service accountsSet 的 `automountServiceAccountToken` 设置为 `false`** + +Kubernetes 提供了一个 `default` service account，用于集群工作负载，在 pod 没有分配特定 service account 时使用。如果需要从 pod 访问 Kubernetes API，则应为该 pod 创建一个特定的 service account，并授予该 service account 权限。`default` service account 应配置为不提供 service account 令牌，并且不具有任何明确的权限分配。 + +对于标准的 RKE2 安装中的每个命名空间，包括 `default` 和 `kube-system`，`default` service account 必须包含此值： + +```yaml +automountServiceAccountToken: false +``` + +对于由集群操作员创建的命名空间，可以使用以下脚本和配置文件来配置 `default` service account。 + +以下配置必须保存到一个名为 `account_update.yaml` 的文件中。 + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: default +automountServiceAccountToken: false +``` + +创建一个名为 `account_update.sh` 的 bash 脚本文件。确保运行 `sudo chmod +x account_update.sh` 命令，以便脚本具有执行权限。 + +```bash +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + echo -n "Patching namespace $namespace - " + kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" +done +``` + +执行此脚本以将 `account_update.yaml` 配置应用到所有命名空间中的 `default` service account。 + +### API server 审计配置 + +CIS 要求 1.2.19 至 1.2.22 与为 API server 配置审计日志有关。当 RKE2 在设置配置文件标志的情况下启动时，它将自动在 API server 中配置加固的 `--audit-log-` 参数以通过这些 CIS 检查。 + +RKE2 的默认审计策略被配置为不记录 API server 中的请求。这样做是为了允许集群操作员灵活地定制符合其审计要求和需求的审计策略，因为这些策略是针对每个用户的环境和政策而特定的。 + +当使用 `profile` 标志启动 RKE2 时，RKE2 会创建一个默认的审计策略。该策略定义在 `/etc/rancher/rke2/audit-policy.yaml` 中。 + +```yaml +apiVersion: audit.k8s.io/v1 +kind: Policy +metadata: + creationTimestamp: null +rules: +- level: None +``` + +## 加固的 RKE2 模板配置参考 + +参考模板配置用于在 Rancher 中创建加固的 RKE2 自定义集群。该参考不包括其他必需的**集群配置**指令，这些指令会根据你的环境而有所不同。 + + + + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: # 定义集群名称 +spec: + defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted + kubernetesVersion: # 定义 RKE2 版本 + rkeConfig: + machineSelectorConfig: + - config: + profile: cis-1.23 +``` + + + + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: # 定义集群名称 +spec: + defaultPodSecurityPolicyTemplateName: restricted-noroot + kubernetesVersion: # 定义 RKE2 版本 + rkeConfig: + machineSelectorConfig: + - config: + profile: cis-1.6 + protect-kernel-defaults: true +``` + + + + +## 结论 + +如果你按照本指南操作，由 Rancher 提供的 RKE2 自定义集群将配置为通过 CIS Kubernetes Benchmark 测试。你可以查看我们的 RKE2 自我评估指南，了解我们是如何验证每个 benchmarks 的，并且你可以在你的集群上执行相同的操作。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md index 83cac7d6e59..05ec2b4aef5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md @@ -1,30 +1,34 @@ --- -title: RKE2 Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.23 +title: RKE2 自我评估指南 - CIS Benchmark v1.23 - K8s v1.23 --- -This document is a companion to the [RKE2 Hardening Guide](../../../../pages-for-subheaders/rke2-hardening-guide.md), which provides prescriptive guidance on how to harden RKE2 clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. + + + -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: +本文档是 [RKE2 加固指南](rke2-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE2 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 -| Rancher Version | CIS Benchmark Version | Kubernetes Version | +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | |-----------------|-----------------------|--------------------| | Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE2 install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE2 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 -This document is for Rancher operators, security teams, auditors and decision makers. +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.23 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 -## Testing Methodology +## 测试方法 -RKE2 launches control plane components as static pods, managed by the kubelet, and uses containerd as the container runtime. Configuration is defined by arguments passed to the container at the time of initialization or via configuration file. +RKE2 将 control plane 组件作为静态 Pod 启动，由 kubelet 管理，并使用 containerd 作为容器运行时。配置是由初始化时或通过配置文件传递给容器的参数定义的。 -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE2 nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE2 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 :::note -This guide only covers `automated` (previously called `scored`) tests. +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md deleted file mode 100644 index ffb7755d7ad..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.24.md +++ /dev/null @@ -1,3196 +0,0 @@ ---- -title: RKE2 Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.24 ---- - -This document is a companion to the [RKE2 Hardening Guide](../../../../pages-for-subheaders/rke2-hardening-guide.md), which provides prescriptive guidance on how to harden RKE2 clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.24 | - -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE2 install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -RKE2 launches control plane components as static pods, managed by the kubelet, and uses containerd as the container runtime. Configuration is defined by arguments passed to the container at the time of initialization or via configuration file. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE2 nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Master Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the -control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %a /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -644 -``` - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' -``` - -**Expected Result**: - -```console -'permissions' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 permissions=644 -``` - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root -``` - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/db/etcd -``` - -**Expected Result**: - -```console -permissions has permissions 700, expected 700 or more restrictive -``` - -**Returned Value**: - -```console -permissions=700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/db/etcd -``` - -**Expected Result**: - -```console -'etcd:etcd' is present -``` - -**Returned Value**: - -```console -etcd:etcd -``` - -### 1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 600 /etc/kubernetes/admin.conf - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/admin.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/admin.conf - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/admin.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 scheduler - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root scheduler - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 controllermanager - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/controller.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /var/lib/rancher/rke2/server/cred/controller.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/controller.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown -R root:root /var/lib/rancher/rke2/server/tls/ - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/tls -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 644 /var/lib/rancher/rke2/server/tls/*.crt - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.crt -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /var/lib/rancher/rke2/server/tls/*.key - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.key -``` - -**Expected Result**: - -```console -'permissions' is equal to '600' -``` - -**Returned Value**: - -```console -permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'AlwaysPullImages' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:34 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 3484 3419 16 23:45 ? 00:01:35 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -### 1.2.33 Ensure that encryption providers are appropriately configured (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit:** - -```bash -/bin/sh -c 'if grep aescbc /var/lib/rancher/rke2/server/cred/encryption-config.json; then echo 0; fi' -``` - -**Expected Result**: - -```console -'0' is present -``` - -**Returned Value**: - -```console -{"kind":"EncryptionConfiguration","apiVersion":"apiserver.config.k8s.io/v1","resources":[{"resources":["secrets"],"providers":[{"aescbc":{"keys":[{"name":"aescbckey","secret":"LdLeqCPN/HfmgnUBXVPkDtUfeOPHwcQzDiLYG3nzFI4="}]}},{"identity":{}}]}]} 0 -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 3645 3538 1 23:45 ? 00:00:10 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3652 3523 0 23:45 ? 00:00:03 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 3652 3523 0 23:45 ? 00:00:03 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-102 ETCD_UNSUPPORTED_ARCH= FILE_HASH=755e8914c854015cfb808b8456f808644387876672add1f9e19564c58f754595 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=78088791fd2612fad13ce760ef91a0ee HOME=/ -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-102 ETCD_UNSUPPORTED_ARCH= FILE_HASH=755e8914c854015cfb808b8456f808644387876672add1f9e19564c58f754595 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=78088791fd2612fad13ce760ef91a0ee HOME=/ -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Audit Config:** - -```bash -cat /var/lib/rancher/rke2/server/db/etcd/config -``` - -**Expected Result**: - -```console -'ETCD_TRUSTED_CA_FILE' is present OR '{.peer-transport-security.trusted-ca-file}' is equal to '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-31-102 ETCD_UNSUPPORTED_ARCH= FILE_HASH=755e8914c854015cfb808b8456f808644387876672add1f9e19564c58f754595 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=78088791fd2612fad13ce760ef91a0ee HOME=/ -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep | grep -o audit-policy-file -``` - -**Expected Result**: - -```console -'audit-policy-file' is equal to 'audit-policy-file' -``` - -**Returned Value**: - -```console -audit-policy-file -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive OR '/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is present OR '/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/kubelet.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit Script:** `check_cafile_permissions.sh` - -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -CAFILE=/node$CAFILE -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi - -``` - -**Audit Execution:** - -```bash -./check_cafile_permissions.sh -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit Script:** `check_cafile_ownership.sh` - -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -CAFILE=/node$CAFILE -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi - -``` - -**Audit Execution:** - -```bash -./check_cafile_ownership.sh -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /etc/rancher/rke2/rke2.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/rancher/rke2/rke2.yaml; then stat -c permissions=%a /etc/rancher/rke2/rke2.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /etc/rancher/rke2/rke2.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/rancher/rke2/rke2.yaml; then stat -c %U:%G /etc/rancher/rke2/rke2.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3221 3096 3 23:44 ? 00:00:19 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-102 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=499dafa1-6719-45d4-a0c6-66109f61b011 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3221 3096 3 23:44 ? 00:00:19 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-102 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=499dafa1-6719-45d4-a0c6-66109f61b011 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3221 3096 3 23:44 ? 00:00:19 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-102 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=499dafa1-6719-45d4-a0c6-66109f61b011 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3221 3096 3 23:44 ? 00:00:19 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-102 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=499dafa1-6719-45d4-a0c6-66109f61b011 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJIZXg2aVo0Q09SYmlTd1pyYmRHa2ZIaHNKb2RZaGRScDFjNklkMngKS05ITHZzZEdxeEc0V2xySVp6WTEwTDlLWDJCVk9Senp0RUZ4cDc3MTlFTnVnNDZqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU1pFWGkwbDNsS3dXS1U3S0x6CkgwZG5aTzVEOWpBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCK2NQUUhLdXBJc05zU3BLVEg2NHJibFpKYTIyZ0oKdlh4TmNxb0dQSEdhUlFJZ1JJdU1hcFVKOHVyVEdxblZGckFldy81aFdOZFdRUjNyN3FXQmlrYnk0aEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRpZ0F3SUJBZ0lJUnIzVlVFbTY4Yll3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU5UQXpOekFlRncweU16QXlNall5TXpRek5UZGFGdzB5TkRBeQpNall5TXpRek5UZGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUTE5SWdKZVJyUldOVGUKZC9UUWcwNWdNcWFMSU9oUU1HSUZKRytTSFJUTVl1RkxBUHhydGozV1g3T3hIRDFDaE5nM2p6VnJKbjcwYitpUApIRzJRNS9XTW8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVViNGJ3aEFWMitabXd0MWlaN2pKMGVDTytGRWd3Q2dZSUtvWkl6ajBFQXdJRFJ3QXcKUkFJZ2ZjNHJWdWJwUWVRMmxiVUJPNkxhK0hKNWVHbUFqSTRHMTV6K1pEZmZqQ1lDSUZGVkFTd1ZzVFRsUXlvMQpicmpsdnJzVW9rRmNxN3RIYjJxZk51dTV2aTB1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxXTnMKYVdWdWRDMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTFqYkdsbGJuUXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLTzZGUG4xd3dqbWFYdzc1T1l1blI1QytMbkpiRlpNYlgyWDRVR0kKMWFKK0tXb0tMZExmRnlZbVdvbHoyV3IzUnBhNGVxRXYySTg1bndINWp1NDZnNXlqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnZodkNFQlhiNW1iQzNXSm51Ck1uUjRJNzRVU0RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCeWR6VWRaMm5IZU56UHY2RkwzYUdabXMrZ2VuTmoKLzNaditGR1dhZi9nK3dJaEFNMnVCVE9maW9vQUptRXBpSi92WUFrQmRMdkdyWEdOY2R6UzdJRzRJR3BXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUQ4MXVnTGVmd0EvekE2U2l0V3czeDNiQWRHQ3A2eXJqYmNSRE14eFgwd3FvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTmZTSUNYa2EwVmpVM25mMDBJTk9ZREttaXlEb1VEQmlCU1J2a2gwVXpHTGhTd0Q4YTdZOQoxbCt6c1J3OVFvVFlONDgxYXlaKzlHL29qeHh0a09mMWpBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3221 3096 3 23:44 ? 00:00:19 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-102 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=499dafa1-6719-45d4-a0c6-66109f61b011 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.makeIPTablesUtilChains}' is present OR '{.makeIPTablesUtilChains}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJIZXg2aVo0Q09SYmlTd1pyYmRHa2ZIaHNKb2RZaGRScDFjNklkMngKS05ITHZzZEdxeEc0V2xySVp6WTEwTDlLWDJCVk9Senp0RUZ4cDc3MTlFTnVnNDZqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU1pFWGkwbDNsS3dXS1U3S0x6CkgwZG5aTzVEOWpBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCK2NQUUhLdXBJc05zU3BLVEg2NHJibFpKYTIyZ0oKdlh4TmNxb0dQSEdhUlFJZ1JJdU1hcFVKOHVyVEdxblZGckFldy81aFdOZFdRUjNyN3FXQmlrYnk0aEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRpZ0F3SUJBZ0lJUnIzVlVFbTY4Yll3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU5UQXpOekFlRncweU16QXlNall5TXpRek5UZGFGdzB5TkRBeQpNall5TXpRek5UZGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUTE5SWdKZVJyUldOVGUKZC9UUWcwNWdNcWFMSU9oUU1HSUZKRytTSFJUTVl1RkxBUHhydGozV1g3T3hIRDFDaE5nM2p6VnJKbjcwYitpUApIRzJRNS9XTW8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVViNGJ3aEFWMitabXd0MWlaN2pKMGVDTytGRWd3Q2dZSUtvWkl6ajBFQXdJRFJ3QXcKUkFJZ2ZjNHJWdWJwUWVRMmxiVUJPNkxhK0hKNWVHbUFqSTRHMTV6K1pEZmZqQ1lDSUZGVkFTd1ZzVFRsUXlvMQpicmpsdnJzVW9rRmNxN3RIYjJxZk51dTV2aTB1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxXTnMKYVdWdWRDMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTFqYkdsbGJuUXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLTzZGUG4xd3dqbWFYdzc1T1l1blI1QytMbkpiRlpNYlgyWDRVR0kKMWFKK0tXb0tMZExmRnlZbVdvbHoyV3IzUnBhNGVxRXYySTg1bndINWp1NDZnNXlqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnZodkNFQlhiNW1iQzNXSm51Ck1uUjRJNzRVU0RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCeWR6VWRaMm5IZU56UHY2RkwzYUdabXMrZ2VuTmoKLzNaditGR1dhZi9nK3dJaEFNMnVCVE9maW9vQUptRXBpSi92WUFrQmRMdkdyWEdOY2R6UzdJRzRJR3BXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUQ4MXVnTGVmd0EvekE2U2l0V3czeDNiQWRHQ3A2eXJqYmNSRE14eFgwd3FvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTmZTSUNYa2EwVmpVM25mMDBJTk9ZREttaXlEb1VEQmlCU1J2a2gwVXpHTGhTd0Q4YTdZOQoxbCt6c1J3OVFvVFlONDgxYXlaKzlHL29qeHh0a09mMWpBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.eventRecordQPS}' is present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJIZXg2aVo0Q09SYmlTd1pyYmRHa2ZIaHNKb2RZaGRScDFjNklkMngKS05ITHZzZEdxeEc0V2xySVp6WTEwTDlLWDJCVk9Senp0RUZ4cDc3MTlFTnVnNDZqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU1pFWGkwbDNsS3dXS1U3S0x6CkgwZG5aTzVEOWpBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCK2NQUUhLdXBJc05zU3BLVEg2NHJibFpKYTIyZ0oKdlh4TmNxb0dQSEdhUlFJZ1JJdU1hcFVKOHVyVEdxblZGckFldy81aFdOZFdRUjNyN3FXQmlrYnk0aEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRpZ0F3SUJBZ0lJUnIzVlVFbTY4Yll3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU5UQXpOekFlRncweU16QXlNall5TXpRek5UZGFGdzB5TkRBeQpNall5TXpRek5UZGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUTE5SWdKZVJyUldOVGUKZC9UUWcwNWdNcWFMSU9oUU1HSUZKRytTSFJUTVl1RkxBUHhydGozV1g3T3hIRDFDaE5nM2p6VnJKbjcwYitpUApIRzJRNS9XTW8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVViNGJ3aEFWMitabXd0MWlaN2pKMGVDTytGRWd3Q2dZSUtvWkl6ajBFQXdJRFJ3QXcKUkFJZ2ZjNHJWdWJwUWVRMmxiVUJPNkxhK0hKNWVHbUFqSTRHMTV6K1pEZmZqQ1lDSUZGVkFTd1ZzVFRsUXlvMQpicmpsdnJzVW9rRmNxN3RIYjJxZk51dTV2aTB1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxXTnMKYVdWdWRDMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTFqYkdsbGJuUXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLTzZGUG4xd3dqbWFYdzc1T1l1blI1QytMbkpiRlpNYlgyWDRVR0kKMWFKK0tXb0tMZExmRnlZbVdvbHoyV3IzUnBhNGVxRXYySTg1bndINWp1NDZnNXlqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnZodkNFQlhiNW1iQzNXSm51Ck1uUjRJNzRVU0RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCeWR6VWRaMm5IZU56UHY2RkwzYUdabXMrZ2VuTmoKLzNaditGR1dhZi9nK3dJaEFNMnVCVE9maW9vQUptRXBpSi92WUFrQmRMdkdyWEdOY2R6UzdJRzRJR3BXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUQ4MXVnTGVmd0EvekE2U2l0V3czeDNiQWRHQ3A2eXJqYmNSRE14eFgwd3FvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTmZTSUNYa2EwVmpVM25mMDBJTk9ZREttaXlEb1VEQmlCU1J2a2gwVXpHTGhTd0Q4YTdZOQoxbCt6c1J3OVFvVFlONDgxYXlaKzlHL29qeHh0a09mMWpBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3221 3096 3 23:44 ? 00:00:19 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-31-102 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=499dafa1-6719-45d4-a0c6-66109f61b011 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJIZXg2aVo0Q09SYmlTd1pyYmRHa2ZIaHNKb2RZaGRScDFjNklkMngKS05ITHZzZEdxeEc0V2xySVp6WTEwTDlLWDJCVk9Senp0RUZ4cDc3MTlFTnVnNDZqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU1pFWGkwbDNsS3dXS1U3S0x6CkgwZG5aTzVEOWpBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCK2NQUUhLdXBJc05zU3BLVEg2NHJibFpKYTIyZ0oKdlh4TmNxb0dQSEdhUlFJZ1JJdU1hcFVKOHVyVEdxblZGckFldy81aFdOZFdRUjNyN3FXQmlrYnk0aEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRpZ0F3SUJBZ0lJUnIzVlVFbTY4Yll3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU5UQXpOekFlRncweU16QXlNall5TXpRek5UZGFGdzB5TkRBeQpNall5TXpRek5UZGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUTE5SWdKZVJyUldOVGUKZC9UUWcwNWdNcWFMSU9oUU1HSUZKRytTSFJUTVl1RkxBUHhydGozV1g3T3hIRDFDaE5nM2p6VnJKbjcwYitpUApIRzJRNS9XTW8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVViNGJ3aEFWMitabXd0MWlaN2pKMGVDTytGRWd3Q2dZSUtvWkl6ajBFQXdJRFJ3QXcKUkFJZ2ZjNHJWdWJwUWVRMmxiVUJPNkxhK0hKNWVHbUFqSTRHMTV6K1pEZmZqQ1lDSUZGVkFTd1ZzVFRsUXlvMQpicmpsdnJzVW9rRmNxN3RIYjJxZk51dTV2aTB1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxXTnMKYVdWdWRDMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTFqYkdsbGJuUXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLTzZGUG4xd3dqbWFYdzc1T1l1blI1QytMbkpiRlpNYlgyWDRVR0kKMWFKK0tXb0tMZExmRnlZbVdvbHoyV3IzUnBhNGVxRXYySTg1bndINWp1NDZnNXlqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnZodkNFQlhiNW1iQzNXSm51Ck1uUjRJNzRVU0RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCeWR6VWRaMm5IZU56UHY2RkwzYUdabXMrZ2VuTmoKLzNaditGR1dhZi9nK3dJaEFNMnVCVE9maW9vQUptRXBpSi92WUFrQmRMdkdyWEdOY2R6UzdJRzRJR3BXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUQ4MXVnTGVmd0EvekE2U2l0V3czeDNiQWRHQ3A2eXJqYmNSRE14eFgwd3FvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTmZTSUNYa2EwVmpVM25mMDBJTk9ZREttaXlEb1VEQmlCU1J2a2gwVXpHTGhTd0Q4YTdZOQoxbCt6c1J3OVFvVFlONDgxYXlaKzlHL29qeHh0a09mMWpBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) - - -**Result:** pass - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.featureGates.RotateKubeletServerCertificate}' is present OR '{.featureGates.RotateKubeletServerCertificate}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJIZXg2aVo0Q09SYmlTd1pyYmRHa2ZIaHNKb2RZaGRScDFjNklkMngKS05ITHZzZEdxeEc0V2xySVp6WTEwTDlLWDJCVk9Senp0RUZ4cDc3MTlFTnVnNDZqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU1pFWGkwbDNsS3dXS1U3S0x6CkgwZG5aTzVEOWpBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCK2NQUUhLdXBJc05zU3BLVEg2NHJibFpKYTIyZ0oKdlh4TmNxb0dQSEdhUlFJZ1JJdU1hcFVKOHVyVEdxblZGckFldy81aFdOZFdRUjNyN3FXQmlrYnk0aEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRpZ0F3SUJBZ0lJUnIzVlVFbTY4Yll3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU5UQXpOekFlRncweU16QXlNall5TXpRek5UZGFGdzB5TkRBeQpNall5TXpRek5UZGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUTE5SWdKZVJyUldOVGUKZC9UUWcwNWdNcWFMSU9oUU1HSUZKRytTSFJUTVl1RkxBUHhydGozV1g3T3hIRDFDaE5nM2p6VnJKbjcwYitpUApIRzJRNS9XTW8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVViNGJ3aEFWMitabXd0MWlaN2pKMGVDTytGRWd3Q2dZSUtvWkl6ajBFQXdJRFJ3QXcKUkFJZ2ZjNHJWdWJwUWVRMmxiVUJPNkxhK0hKNWVHbUFqSTRHMTV6K1pEZmZqQ1lDSUZGVkFTd1ZzVFRsUXlvMQpicmpsdnJzVW9rRmNxN3RIYjJxZk51dTV2aTB1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxXTnMKYVdWdWRDMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTFqYkdsbGJuUXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLTzZGUG4xd3dqbWFYdzc1T1l1blI1QytMbkpiRlpNYlgyWDRVR0kKMWFKK0tXb0tMZExmRnlZbVdvbHoyV3IzUnBhNGVxRXYySTg1bndINWp1NDZnNXlqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnZodkNFQlhiNW1iQzNXSm51Ck1uUjRJNzRVU0RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCeWR6VWRaMm5IZU56UHY2RkwzYUdabXMrZ2VuTmoKLzNaditGR1dhZi9nK3dJaEFNMnVCVE9maW9vQUptRXBpSi92WUFrQmRMdkdyWEdOY2R6UzdJRzRJR3BXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUQ4MXVnTGVmd0EvekE2U2l0V3czeDNiQWRHQ3A2eXJqYmNSRE14eFgwd3FvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTmZTSUNYa2EwVmpVM25mMDBJTk9ZREttaXlEb1VEQmlCU1J2a2gwVXpHTGhTd0Q4YTdZOQoxbCt6c1J3OVFvVFlONDgxYXlaKzlHL29qeHh0a09mMWpBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{range .tlsCipherSuites[:]}{}{','}{end}' is present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlRENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJIZXg2aVo0Q09SYmlTd1pyYmRHa2ZIaHNKb2RZaGRScDFjNklkMngKS05ITHZzZEdxeEc0V2xySVp6WTEwTDlLWDJCVk9Senp0RUZ4cDc3MTlFTnVnNDZqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCU1pFWGkwbDNsS3dXS1U3S0x6CkgwZG5aTzVEOWpBS0JnZ3Foa2pPUFFRREFnTkhBREJFQWlCK2NQUUhLdXBJc05zU3BLVEg2NHJibFpKYTIyZ0oKdlh4TmNxb0dQSEdhUlFJZ1JJdU1hcFVKOHVyVEdxblZGckFldy81aFdOZFdRUjNyN3FXQmlrYnk0aEk9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRpZ0F3SUJBZ0lJUnIzVlVFbTY4Yll3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU5UQXpOekFlRncweU16QXlNall5TXpRek5UZGFGdzB5TkRBeQpNall5TXpRek5UZGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUTE5SWdKZVJyUldOVGUKZC9UUWcwNWdNcWFMSU9oUU1HSUZKRytTSFJUTVl1RkxBUHhydGozV1g3T3hIRDFDaE5nM2p6VnJKbjcwYitpUApIRzJRNS9XTW8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVViNGJ3aEFWMitabXd0MWlaN2pKMGVDTytGRWd3Q2dZSUtvWkl6ajBFQXdJRFJ3QXcKUkFJZ2ZjNHJWdWJwUWVRMmxiVUJPNkxhK0hKNWVHbUFqSTRHMTV6K1pEZmZqQ1lDSUZGVkFTd1ZzVFRsUXlvMQpicmpsdnJzVW9rRmNxN3RIYjJxZk51dTV2aTB1Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxXTnMKYVdWdWRDMWpZVUF4TmpjM05EVTFNRE0zTUI0WERUSXpNREl5TmpJek5ETTFOMW9YRFRNek1ESXlNekl6TkRNMQpOMW93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTFqYkdsbGJuUXRZMkZBTVRZM056UTFOVEF6TnpCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJLTzZGUG4xd3dqbWFYdzc1T1l1blI1QytMbkpiRlpNYlgyWDRVR0kKMWFKK0tXb0tMZExmRnlZbVdvbHoyV3IzUnBhNGVxRXYySTg1bndINWp1NDZnNXlqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnZodkNFQlhiNW1iQzNXSm51Ck1uUjRJNzRVU0RBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlCeWR6VWRaMm5IZU56UHY2RkwzYUdabXMrZ2VuTmoKLzNaditGR1dhZi9nK3dJaEFNMnVCVE9maW9vQUptRXBpSi92WUFrQmRMdkdyWEdOY2R6UzdJRzRJR3BXCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUQ4MXVnTGVmd0EvekE2U2l0V3czeDNiQWRHQ3A2eXJqYmNSRE14eFgwd3FvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTmZTSUNYa2EwVmpVM25mMDBJTk9ZREttaXlEb1VEQmlCU1J2a2gwVXpHTGhTd0Q4YTdZOQoxbCt6c1J3OVFvVFlONDgxYXlaKzlHL29qeHh0a09mMWpBPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -**Audit Script:** `check_for_default_sa.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` - -**Audit Execution:** - -```bash -./check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): serviceaccounts is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "serviceaccounts" in API group "" at the cluster scope Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "calico-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-fleet-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-impersonation-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cis-operator-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-node-lease" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-public" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "local" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "tigera-operator" true -``` - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -**Audit:** - -```bash -kubectl get psp global-restricted-psp -o json | jq -r '.spec.runAsUser.rule' -``` - -**Expected Result**: - -```console -'MustRunAsNonRoot' is present -``` - -**Returned Value**: - -```console -MustRunAsNonRoot -``` - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=5 -``` - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=5 -``` - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=2 -``` - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=4 -``` - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** pass - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=4 -``` - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** pass - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -**Audit:** - -```bash -kubectl get psp global-restricted-psp -o json | jq -r .spec.requiredDropCapabilities[] -``` - -**Expected Result**: - -```console -'ALL' is present -``` - -**Returned Value**: - -```console -ALL -``` - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports Network Policies (Automated) - - -**Result:** pass - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -**Audit:** - -```bash -kubectl get pods --all-namespaces --selector='k8s-app in (calico-node, canal, cilium)' -o name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -**Audit Script:** `check_for_rke2_network_policies.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in kube-system kube-public default; do - policy_count=$(/var/lib/rancher/rke2/bin/kubectl get networkpolicy -n ${namespace} -o json | jq -r '.items | length') - if [ ${policy_count} -eq 0 ]; then - echo "false" - exit - fi -done - -echo "true" - -``` - -**Audit Execution:** - -```bash -./check_for_rke2_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is present -``` - -**Returned Value**: - -```console -true -``` - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md deleted file mode 100644 index 4d524d3fab4..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.25.md +++ /dev/null @@ -1,3196 +0,0 @@ ---- -title: RKE2 Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.25 ---- - -This document is a companion to the [RKE2 Hardening Guide](../../../../pages-for-subheaders/rke2-hardening-guide.md), which provides prescriptive guidance on how to harden RKE2 clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.25 | - -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE2 install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -RKE2 launches control plane components as static pods, managed by the kubelet, and uses containerd as the container runtime. Configuration is defined by arguments passed to the container at the time of initialization or via configuration file. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE2 nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Master Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the -control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %a /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -644 -``` - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' -``` - -**Expected Result**: - -```console -'permissions' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 permissions=644 -``` - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root -``` - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/db/etcd -``` - -**Expected Result**: - -```console -permissions has permissions 700, expected 700 or more restrictive -``` - -**Returned Value**: - -```console -permissions=700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/db/etcd -``` - -**Expected Result**: - -```console -'etcd:etcd' is present -``` - -**Returned Value**: - -```console -etcd:etcd -``` - -### 1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 600 /etc/kubernetes/admin.conf - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/admin.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/admin.conf - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/admin.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 scheduler - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root scheduler - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 controllermanager - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/controller.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /var/lib/rancher/rke2/server/cred/controller.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/controller.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown -R root:root /var/lib/rancher/rke2/server/tls/ - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/tls -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 644 /var/lib/rancher/rke2/server/tls/*.crt - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.crt -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /var/lib/rancher/rke2/server/tls/*.key - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.key -``` - -**Expected Result**: - -```console -'permissions' is equal to '600' -``` - -**Returned Value**: - -```console -permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'AlwaysPullImages' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true root 15378 3910 99 23:32 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/readyz -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -### 1.2.33 Ensure that encryption providers are appropriately configured (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit:** - -```bash -/bin/sh -c 'if grep aescbc /var/lib/rancher/rke2/server/cred/encryption-config.json; then echo 0; fi' -``` - -**Expected Result**: - -```console -'0' is present -``` - -**Returned Value**: - -```console -{"kind":"EncryptionConfiguration","apiVersion":"apiserver.config.k8s.io/v1","resources":[{"resources":["secrets"],"providers":[{"aescbc":{"keys":[{"name":"aescbckey","secret":"TSpBkJhIU0sRx+84IZuBZ1qO+eaRdW31C7QCnF3+n8s="}]}},{"identity":{}}]}]} 0 -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4126 4014 0 23:27 ? 00:00:02 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 4126 4014 0 23:27 ? 00:00:02 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-25-112 ETCD_UNSUPPORTED_ARCH= POD_HASH=ab0b8a2ee7711940d3d951edece075f3 FILE_HASH=068666c5f959fc1023cb1761daaaed212727c04120747d825c78fd7683122e6d NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 HOME=/ -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-25-112 ETCD_UNSUPPORTED_ARCH= POD_HASH=ab0b8a2ee7711940d3d951edece075f3 FILE_HASH=068666c5f959fc1023cb1761daaaed212727c04120747d825c78fd7683122e6d NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 HOME=/ -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Audit Config:** - -```bash -cat /var/lib/rancher/rke2/server/db/etcd/config -``` - -**Expected Result**: - -```console -'ETCD_TRUSTED_CA_FILE' is present OR '{.peer-transport-security.trusted-ca-file}' is equal to '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-25-112 ETCD_UNSUPPORTED_ARCH= POD_HASH=ab0b8a2ee7711940d3d951edece075f3 FILE_HASH=068666c5f959fc1023cb1761daaaed212727c04120747d825c78fd7683122e6d NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 HOME=/ -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep | grep -o audit-policy-file -``` - -**Expected Result**: - -```console -'audit-policy-file' is equal to 'audit-policy-file' -``` - -**Returned Value**: - -```console -audit-policy-file -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive OR '/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is present OR '/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/kubelet.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit Script:** `check_cafile_permissions.sh` - -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -CAFILE=/node$CAFILE -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi - -``` - -**Audit Execution:** - -```bash -./check_cafile_permissions.sh -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit Script:** `check_cafile_ownership.sh` - -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -CAFILE=/node$CAFILE -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi - -``` - -**Audit Execution:** - -```bash -./check_cafile_ownership.sh -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /etc/rancher/rke2/rke2.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/rancher/rke2/rke2.yaml; then stat -c permissions=%a /etc/rancher/rke2/rke2.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /etc/rancher/rke2/rke2.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/rancher/rke2/rke2.yaml; then stat -c %U:%G /etc/rancher/rke2/rke2.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.makeIPTablesUtilChains}' is present OR '{.makeIPTablesUtilChains}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.eventRecordQPS}' is present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) - - -**Result:** pass - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.featureGates.RotateKubeletServerCertificate}' is present OR '{.featureGates.RotateKubeletServerCertificate}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{range .tlsCipherSuites[:]}{}{','}{end}' is present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -**Audit Script:** `check_for_default_sa.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` - -**Audit Execution:** - -```bash -./check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): serviceaccounts is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "serviceaccounts" in API group "" at the cluster scope Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "calico-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-fleet-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-impersonation-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cis-operator-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-node-lease" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-public" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "local" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "tigera-operator" true -``` - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -**Audit:** - -```bash -kubectl get psp global-restricted-psp -o json | jq -r '.spec.runAsUser.rule' -``` - -**Expected Result**: - -```console -'MustRunAsNonRoot' is present -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" -``` - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** fail - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -**Audit:** - -```bash -kubectl get psp global-restricted-psp -o json | jq -r .spec.requiredDropCapabilities[] -``` - -**Expected Result**: - -```console -'ALL' is present -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" -``` - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports Network Policies (Automated) - - -**Result:** pass - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -**Audit:** - -```bash -kubectl get pods --all-namespaces --selector='k8s-app in (calico-node, canal, cilium)' -o name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - - -**Result:** true - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -**Audit Script:** `check_for_rke2_network_policies.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in kube-system kube-public default; do - policy_count=$(/var/lib/rancher/rke2/bin/kubectl get networkpolicy -n ${namespace} -o json | jq -r '.items | length') - if [ ${policy_count} -eq 0 ]; then - echo "false" - exit - fi -done - -echo "true" - -``` - -**Audit Execution:** - -```bash -./check_for_rke2_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is present -``` - -**Returned Value**: - -```console -true -``` - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md new file mode 100644 index 00000000000..f14207e929d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md @@ -0,0 +1,3202 @@ +--- +title: RKE2 自我评估指南 - CIS Benchmark v1.24 - K8s v1.24 +--- + + + + + +本文档是 [RKE2 加固指南](rke2-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE2 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | + +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE2 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.24 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +RKE2 将 control plane 组件作为静态 Pod 启动，由 kubelet 管理，并使用 containerd 作为容器运行时。配置是由初始化时或通过配置文件传递给容器的参数定义的。 + +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE2 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 644 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %a /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' +``` + +**Expected Result**: + +```console +'600' is present +``` + +**Returned Value**: + +```console +644 +``` + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' +``` + +**Expected Result**: + +```console +'permissions' is equal to '600' +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' +``` + +**Expected Result**: + +```console +'644' is equal to '644' +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=644 +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root root:root +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/db/etcd +``` + +**Expected Result**: + +```console +permissions has permissions 700, expected 700 or more restrictive +``` + +**Returned Value**: + +```console +permissions=700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/db/etcd +``` + +**Expected Result**: + +```console +'etcd:etcd' is present +``` + +**Returned Value**: + +```console +etcd:etcd +``` + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/admin.conf + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/cred/admin.kubeconfig +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/admin.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 scheduler + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root scheduler + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 controllermanager + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/cred/controller.kubeconfig +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/server/cred/controller.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/controller.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /var/lib/rancher/rke2/server/tls/ + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/tls +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Automated) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 644 /var/lib/rancher/rke2/server/tls/*.crt + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.crt +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /var/lib/rancher/rke2/server/tls/*.key + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.key +``` + +**Expected Result**: + +```console +'permissions' is equal to '600' +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the --kubelet-https parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-https' is present OR '--kubelet-https' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.18 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxsize' is greater or equal to 100 +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true root 1017917 2489 83 16:16 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/readyz root 1017945 2489 75 16:16 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/livez +``` + +### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +### 1.2.33 Ensure that encryption providers are appropriately configured (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit:** + +```bash +/bin/sh -c 'if grep aescbc /var/lib/rancher/rke2/server/cred/encryption-config.json; then echo 0; fi' +``` + +**Expected Result**: + +```console +'0' is present +``` + +**Returned Value**: + +```console +{"kind":"EncryptionConfiguration","apiVersion":"apiserver.config.k8s.io/v1","resources":[{"resources":["secrets"],"providers":[{"aescbc":{"keys":[{"name":"aescbckey","secret":"akmMjUAq94YvsUftJpiA3b+9SClu0ESPBeckAI7KZBY="}]}},{"identity":{}}]}]} 0 +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2707 2593 0 Sep11 ? 00:06:20 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 2707 2593 0 Sep11 ? 00:06:20 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-10-113 ETCD_UNSUPPORTED_ARCH= NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=560b915d9afb672c20c2c1e8664bdf8f FILE_HASH=166aebdce42ff62fcdd2cefc9e7ed8f7c5b562d219ca6afec8f73adc654f65e7 HOME=/ +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-10-113 ETCD_UNSUPPORTED_ARCH= NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=560b915d9afb672c20c2c1e8664bdf8f FILE_HASH=166aebdce42ff62fcdd2cefc9e7ed8f7c5b562d219ca6afec8f73adc654f65e7 HOME=/ +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Audit Config:** + +```bash +cat /var/lib/rancher/rke2/server/db/etcd/config +``` + +**Expected Result**: + +```console +'ETCD_TRUSTED_CA_FILE' is present OR '{.peer-transport-security.trusted-ca-file}' is equal to '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-10-113 ETCD_UNSUPPORTED_ARCH= NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=560b915d9afb672c20c2c1e8664bdf8f FILE_HASH=166aebdce42ff62fcdd2cefc9e7ed8f7c5b562d219ca6afec8f73adc654f65e7 HOME=/ +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep | grep -o audit-policy-file +``` + +**Expected Result**: + +```console +'audit-policy-file' is equal to 'audit-policy-file' +``` + +**Returned Value**: + +```console +audit-policy-file +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'600' is present +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit Script:** `check_cafile_permissions.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +CAFILE=/node$CAFILE +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_permissions.sh +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit Script:** `check_cafile_ownership.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +CAFILE=/node$CAFILE +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_ownership.sh +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--protect-kernel-defaults' is equal to 'true' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.makeIPTablesUtilChains}' is present OR '{.makeIPTablesUtilChains}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +### 4.2.9 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Automated) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.eventRecordQPS}' is present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** pass + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.featureGates.RotateKubeletServerCertificate}' is present OR '{.featureGates.RotateKubeletServerCertificate}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{range .tlsCipherSuites[:]}{}{','}{end}' is present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Automated) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[]?.kind=="ServiceAccount" and .subjects[]?.name=="default") or (.subjects[]?.kind=="Group" and .subjects[]?.name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[]? != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +**Audit:** + +```bash +kubectl get psp global-restricted-psp && kubectl get psp global-restricted-psp -o json | jq -r ".spec.runAsUser.rule" || kubectl get psp restricted-noroot-psp && kubectl get psp restricted-noroot-psp -o json | jq -r ".spec.runAsUser.rule" +``` + +**Expected Result**: + +```console +'MustRunAsNonRoot' is equal to 'MustRunAsNonRoot' +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ Error from server (NotFound): podsecuritypolicies.policy "global-restricted-psp" not found Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES restricted-noroot-psp false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,csi,persistentVolumeClaim Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ MustRunAsNonRoot +``` + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=5 +``` + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=5 +``` + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=2 +``` + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=4 +``` + +### 5.2.7 Minimize the admission of root containers (Automated) + + +**Result:** pass + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=4 +``` + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +**Audit:** + +```bash +kubectl get psp global-restricted-psp && kubectl get psp global-restricted-psp -o json | jq -r .spec.requiredDropCapabilities[] || kubectl get psp restricted-noroot-psp && kubectl get psp restricted-noroot-psp -o json | jq -r .spec.requiredDropCapabilities[] +``` + +**Expected Result**: + +```console +'ALL' is equal to 'ALL' +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ Error from server (NotFound): podsecuritypolicies.policy "global-restricted-psp" not found Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES restricted-noroot-psp false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,csi,persistentVolumeClaim Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ ALL +``` + +### 5.2.9 Minimize the admission of containers with added capabilities (Automated) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports Network Policies (Automated) + + +**Result:** pass + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +**Audit:** + +```bash +kubectl get pods --all-namespaces --selector='k8s-app in (calico-node, canal, cilium)' -o name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +--count=1 +``` + +### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +**Audit Script:** `check_for_rke2_network_policies.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +for namespace in kube-system kube-public default; do + policy_count=$(/var/lib/rancher/rke2/bin/kubectl get networkpolicy -n ${namespace} -o json | jq -r '.items | length') + if [ ${policy_count} -eq 0 ]; then + echo "false" + exit + fi +done + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_rke2_network_policies.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md new file mode 100644 index 00000000000..a51c617eea8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md @@ -0,0 +1,2967 @@ +--- +title: RKE2 自我评估指南 - CIS Benchmark v1.7 - K8s v1.25/v1.26/v1.27 +--- + + + + + +本文档是 [RKE2 加固指南](rke2-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE2 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25/v1.26/v1.27 | + +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE2 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.7 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +RKE2 将 control plane 组件作为静态 Pod 启动，由 kubelet 管理，并使用 containerd 作为容器运行时。配置是由初始化时或通过配置文件传递给容器的参数定义的。 + +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE2 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then find /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml -name '*etcd*' | xargs stat -c permissions=%a; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=644 +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root root:root +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/db/etcd +``` + +**Expected Result**: + +```console +permissions has permissions 700, expected 700 or more restrictive +``` + +**Returned Value**: + +```console +permissions=700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** Not Applicable + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/admin.conf + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/cred/admin.kubeconfig +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/admin.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'600' is present +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /var/lib/rancher/rke2/server/cred/controller.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/server/cred/controller.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'600' is present +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/server/cred/controller.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/controller.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/tls +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /var/lib/rancher/rke2/server/tls/*.crt + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.crt +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /var/lib/rancher/rke2/server/tls/*.key + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.key +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.7 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.9 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.12 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.13 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true root 1018768 2419 99 16:17 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/readyz +``` + +### 1.2.14 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.15 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.16 Ensure that the --secure-port argument is not set to 0 - NoteThis recommendation is obsolete and will be deleted per the consensus process (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.17 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.18 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.19 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.20 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.21 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +### 1.2.22 Ensure that the --request-timeout argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, --request-timeout=300s + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--request-timeout' is not present OR '--request-timeout' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.23 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.24 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.25 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.26 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.27 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.28 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.29 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.30 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit:** + +```bash +ENCRYPTION_PROVIDER_CONFIG=$(ps -ef | grep kube-apiserver | grep -- --encryption-provider-config | sed 's%.*encryption-provider-config[= ]\([^ ]*\).*%\1%') if test -e $ENCRYPTION_PROVIDER_CONFIG; then grep -A1 'providers:' $ENCRYPTION_PROVIDER_CONFIG | tail -n1 | grep -o "[A-Za-z]*" | sed 's/^/provider=/'; fi +``` + +**Expected Result**: + +```console +'provider' is present +``` + +### 1.2.31 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 +Not Applicable. + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2645 2538 0 Sep11 ? 00:05:26 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 2645 2538 0 Sep11 ? 00:05:26 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= +Not Applicable. + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" +Not Applicable. + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-14-226 ETCD_UNSUPPORTED_ARCH= FILE_HASH=e9ca6f328e70dd3c17ba78c302ee32927a4961198e95d1d948114a5d7e350d99 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=aa1658a59ab75324ef67c786d08307e3 HOME=/ +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= +Not Applicable. + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true +Not Applicable. + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-14-226 ETCD_UNSUPPORTED_ARCH= FILE_HASH=e9ca6f328e70dd3c17ba78c302ee32927a4961198e95d1d948114a5d7e350d99 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=aa1658a59ab75324ef67c786d08307e3 HOME=/ +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Audit Config:** + +```bash +cat /var/lib/rancher/rke2/server/db/etcd/config +``` + +**Expected Result**: + +```console +'ETCD_TRUSTED_CA_FILE' is present OR '{.peer-transport-security.trusted-ca-file}' is equal to '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-14-226 ETCD_UNSUPPORTED_ARCH= FILE_HASH=e9ca6f328e70dd3c17ba78c302ee32927a4961198e95d1d948114a5d7e350d99 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=aa1658a59ab75324ef67c786d08307e3 HOME=/ +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +### 3.1.2 Service account token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of service account tokens. + +### 3.1.3 Bootstrap token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of bootstrap tokens. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not applicable. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit Script:** `check_cafile_permissions.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +CAFILE=/node$CAFILE +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_permissions.sh +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit Script:** `check_cafile_ownership.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +CAFILE=/node$CAFILE +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_ownership.sh +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.makeIPTablesUtilChains}' is present OR '{.makeIPTablesUtilChains}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.7 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable. + +### 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.eventRecordQPS}' is present OR '{.eventRecordQPS}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.10 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** pass + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.featureGates.RotateKubeletServerCertificate}' is present OR '{.featureGates.RotateKubeletServerCertificate}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{range .tlsCipherSuites[:]}{}{','}{end}' is present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.13 Ensure that a limit is set on pod PIDs (Manual) + + +**Result:** warn + +**Remediation:** +Decide on an appropriate level for this parameter and set it, +either via the --pod-max-pids command line parameter or the PodPidsLimit configuration file setting. + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.podPidsLimit}' is present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** warn + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +### 5.1.9 Minimize access to create persistent volumes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to PersistentVolume objects in the cluster. + +### 5.1.10 Minimize access to the proxy sub-resource of nodes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the proxy sub-resource of node objects. + +### 5.1.11 Minimize access to the approval sub-resource of certificatesigningrequests objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the approval sub-resource of certificatesigningrequest objects. + +### 5.1.12 Minimize access to webhook configuration objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the validatingwebhookconfigurations or mutatingwebhookconfigurations objects + +### 5.1.13 Minimize access to the service account token creation (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the token sub-resource of serviceaccount objects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +### 5.2.7 Minimize the admission of root containers (Manual) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Automated) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md deleted file mode 100644 index 252accbb79d..00000000000 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25.md +++ /dev/null @@ -1,3196 +0,0 @@ ---- -title: RKE2 Self-Assessment Guide - CIS Benchmark v1.7 - K8s v1.25 ---- - -This document is a companion to the [RKE2 Hardening Guide](../../../../pages-for-subheaders/rke2-hardening-guide.md), which provides prescriptive guidance on how to harden RKE2 clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. - -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: - -| Rancher Version | CIS Benchmark Version | Kubernetes Version | -|-----------------|-----------------------|--------------------| -| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 | - -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE2 install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. - -This document is for Rancher operators, security teams, auditors and decision makers. - -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.7. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). - -## Testing Methodology - -RKE2 launches control plane components as static pods, managed by the kubelet, and uses containerd as the container runtime. Configuration is defined by arguments passed to the container at the time of initialization or via configuration file. - -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE2 nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. - -:::note - -This guide only covers `automated` (previously called `scored`) tests. - -::: - -### Controls - -## 1.1 Master Node Configuration Files -### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the -control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %a /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -644 -``` - -### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' -``` - -**Expected Result**: - -```console -'permissions' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 644 - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 permissions=644 -``` - -### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root - -**Audit:** - -```bash -ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root root:root -``` - -### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). For example, -chmod 700 /var/lib/etcd - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/db/etcd -``` - -**Expected Result**: - -```console -permissions has permissions 700, expected 700 or more restrictive -``` - -**Returned Value**: - -```console -permissions=700 -``` - -### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) - - -**Result:** pass - -**Remediation:** -On the etcd server node, get the etcd data directory, passed as an argument --data-dir, -from the command 'ps -ef | grep etcd'. -Run the below command (based on the etcd data directory found above). -For example, chown etcd:etcd /var/lib/etcd - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/db/etcd -``` - -**Expected Result**: - -```console -'etcd:etcd' is present -``` - -**Returned Value**: - -```console -etcd:etcd -``` - -### 1.1.13 Ensure that the admin.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chmod 600 /etc/kubernetes/admin.conf - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/admin.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, chown root:root /etc/kubernetes/admin.conf - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/admin.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.15 Ensure that the scheduler.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 scheduler - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root scheduler - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod 644 controllermanager - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/cred/controller.kubeconfig -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown root:root /var/lib/rancher/rke2/server/cred/controller.kubeconfig - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/cred/controller.kubeconfig -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chown -R root:root /var/lib/rancher/rke2/server/tls/ - -**Audit:** - -```bash -stat -c %U:%G /var/lib/rancher/rke2/server/tls -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 644 /var/lib/rancher/rke2/server/tls/*.crt - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.crt -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 -``` - -### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the control plane node. -For example, -chmod -R 600 /var/lib/rancher/rke2/server/tls/*.key - -**Audit:** - -```bash -stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.key -``` - -**Expected Result**: - -```console -'permissions' is equal to '600' -``` - -**Returned Value**: - -```console -permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 -``` - -## 1.2 API Server -### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---anonymous-auth=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and configure alternate mechanisms for authentication. Then, -edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the --token-auth-file= parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--token-auth-file' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the `DenyServiceExternalIPs` -from enabled admission plugins. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and remove the --kubelet-https parameter. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-https' is present OR '--kubelet-https' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the -apiserver and kubelets. Then, edit API server pod specification file -/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the -kubelet client certificate and key parameters as below. ---kubelet-client-certificate= ---kubelet-client-key= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and setup the TLS connection between -the apiserver and kubelets. Then, edit the API server pod specification file -/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the ---kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. ---kubelet-certificate-authority= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--kubelet-certificate-authority' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. -One such example could be as below. ---authorization-mode=RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes Node. ---authorization-mode=Node,RBAC - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'Node' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, -for example `--authorization-mode=Node,RBAC`. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--authorization-mode' has 'RBAC' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and set the desired limits in a configuration file. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -and set the below parameters. ---enable-admission-plugins=...,EventRateLimit,... ---admission-control-config-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'EventRateLimit' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a -value that does not include AlwaysAdmit. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -AlwaysPullImages. ---enable-admission-plugins=...,AlwaysPullImages,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'AlwaysPullImages' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) - - -**Result:** warn - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to include -SecurityContextDeny, unless PodSecurityPolicy is already in place. ---enable-admission-plugins=...,SecurityContextDeny,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:05 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the documentation and create ServiceAccount objects as per your environment. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and ensure that the --disable-admission-plugins parameter is set to a -value that does not include ServiceAccount. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --disable-admission-plugins parameter to -ensure it does not include NamespaceLifecycle. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true root 15378 3910 99 23:32 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/readyz -``` - -### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --enable-admission-plugins parameter to a -value that includes NodeRestriction. ---enable-admission-plugins=...,NodeRestriction,... - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--enable-admission-plugins' has 'NodeRestriction' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and either remove the --secure-port parameter or -set it to a different (non-zero) desired port. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--secure-port' is greater than 0 OR '--secure-port' is not present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.18 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-path parameter to a suitable path and -file where you would like audit logs to be written, for example, ---audit-log-path=/var/log/apiserver/audit.log - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-path' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxage parameter to 30 -or as an appropriate number of days, for example, ---audit-log-maxage=30 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxage' is greater or equal to 30 -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate -value. For example, ---audit-log-maxbackup=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxbackup' is greater or equal to 10 -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. -For example, to set it as 100 MB, --audit-log-maxsize=100 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--audit-log-maxsize' is greater or equal to 100 -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---service-account-lookup=true -Alternatively, you can delete the --service-account-lookup parameter from this file so -that the default takes effect. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-lookup' is not present OR '--service-account-lookup' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --service-account-key-file parameter -to the public key file for service accounts. For example, ---service-account-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-key-file' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate and key file parameters. ---etcd-certfile= ---etcd-keyfile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-certfile' is present AND '--etcd-keyfile' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the TLS certificate and private key file parameters. ---tls-cert-file= ---tls-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection on the apiserver. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the client certificate authority file. ---client-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the etcd certificate authority file parameter. ---etcd-cafile= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--etcd-cafile' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml -on the control plane node and set the --encryption-provider-config parameter to the path of that file. -For example, --encryption-provider-config= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep -``` - -**Expected Result**: - -```console -'--encryption-provider-config' is present -``` - -**Returned Value**: - -```console -root 3980 3910 19 23:26 ? 00:01:06 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml -on the control plane node and set the below parameter. ---tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, -TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, -TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, -TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, -TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, -TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 - -### 1.2.33 Ensure that encryption providers are appropriately configured (Automated) - - -**Result:** pass - -**Remediation:** -Follow the Kubernetes documentation and configure a EncryptionConfig file. -In this file, choose aescbc, kms or secretbox as the encryption provider. - -**Audit:** - -```bash -/bin/sh -c 'if grep aescbc /var/lib/rancher/rke2/server/cred/encryption-config.json; then echo 0; fi' -``` - -**Expected Result**: - -```console -'0' is present -``` - -**Returned Value**: - -```console -{"kind":"EncryptionConfiguration","apiVersion":"apiserver.config.k8s.io/v1","resources":[{"resources":["secrets"],"providers":[{"aescbc":{"keys":[{"name":"aescbckey","secret":"TSpBkJhIU0sRx+84IZuBZ1qO+eaRdW31C7QCnF3+n8s="}]}},{"identity":{}}]}]} 0 -``` - -## 1.3 Controller Manager -### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, -for example, --terminated-pod-gc-threshold=10 - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--terminated-pod-gc-threshold' is present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.2 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node to set the below parameter. ---use-service-account-credentials=true - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--use-service-account-credentials' is not equal to 'false' -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --service-account-private-key-file parameter -to the private key file for service accounts. ---service-account-private-key-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--service-account-private-key-file' is present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. ---root-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--root-ca-file' is present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. ---feature-gates=RotateKubeletServerCertificate=true - -### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-controller-manager | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 4128 4029 2 23:27 ? 00:00:06 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true -``` - -## 1.4 Scheduler -### 1.4.1 Ensure that the --profiling argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml file -on the control plane node and set the below parameter. ---profiling=false - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--profiling' is equal to 'false' -``` - -**Returned Value**: - -```console -root 4126 4014 0 23:27 ? 00:00:02 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 -``` - -### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) - - -**Result:** pass - -**Remediation:** -Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml -on the control plane node and ensure the correct value for the --bind-address parameter - -**Audit:** - -```bash -/bin/ps -ef | grep kube-scheduler | grep -v grep -``` - -**Expected Result**: - -```console -'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present -``` - -**Returned Value**: - -```console -root 4126 4014 0 23:27 ? 00:00:02 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 -``` - -## 2 Etcd Node Configuration -### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Follow the etcd service documentation and configure TLS encryption. -Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml -on the master node and set the below parameters. ---cert-file= ---key-file= - -### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and set the below parameter. ---client-cert-auth="true" - -### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and either remove the --auto-tls parameter or set it to false. ---auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-25-112 ETCD_UNSUPPORTED_ARCH= POD_HASH=ab0b8a2ee7711940d3d951edece075f3 FILE_HASH=068666c5f959fc1023cb1761daaaed212727c04120747d825c78fd7683122e6d NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 HOME=/ -``` - -### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Follow the etcd service documentation and configure peer TLS encryption as appropriate -for your etcd cluster. -Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the -master node and set the below parameters. ---peer-client-file= ---peer-key-file= - -### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and set the below parameter. ---peer-client-cert-auth=true - -### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) - - -**Result:** pass - -**Remediation:** -Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master -node and either remove the --peer-auto-tls parameter or set it to false. ---peer-auto-tls=false - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Expected Result**: - -```console -'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-25-112 ETCD_UNSUPPORTED_ARCH= POD_HASH=ab0b8a2ee7711940d3d951edece075f3 FILE_HASH=068666c5f959fc1023cb1761daaaed212727c04120747d825c78fd7683122e6d NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 HOME=/ -``` - -### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) - - -**Result:** pass - -**Remediation:** -[Manual test] -Follow the etcd documentation and create a dedicated certificate authority setup for the -etcd service. -Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the -master node and set the below parameter. ---trusted-ca-file= - -**Audit:** - -```bash -/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep -``` - -**Audit Config:** - -```bash -cat /var/lib/rancher/rke2/server/db/etcd/config -``` - -**Expected Result**: - -```console -'ETCD_TRUSTED_CA_FILE' is present OR '{.peer-transport-security.trusted-ca-file}' is equal to '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' -``` - -**Returned Value**: - -```console -PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-25-112 ETCD_UNSUPPORTED_ARCH= POD_HASH=ab0b8a2ee7711940d3d951edece075f3 FILE_HASH=068666c5f959fc1023cb1761daaaed212727c04120747d825c78fd7683122e6d NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 HOME=/ -``` - -## 3.1 Authentication and Authorization -### 3.1.1 Client certificate authentication should not be used for users (Manual) - - -**Result:** warn - -**Remediation:** -Alternative mechanisms provided by Kubernetes such as the use of OIDC should be -implemented in place of client certificates. - -## 3.2 Logging -### 3.2.1 Ensure that a minimal audit policy is created (Automated) - - -**Result:** pass - -**Remediation:** -Create an audit policy file for your cluster. - -**Audit:** - -```bash -/bin/ps -ef | grep kube-apiserver | grep -v grep | grep -o audit-policy-file -``` - -**Expected Result**: - -```console -'audit-policy-file' is equal to 'audit-policy-file' -``` - -**Returned Value**: - -```console -audit-policy-file -``` - -### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) - - -**Result:** warn - -**Remediation:** -Review the audit policy provided for the cluster and ensure that it covers -at least the following areas, -- Access to Secrets managed by the cluster. Care should be taken to only - log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in - order to avoid risk of logging sensitive data. -- Modification of Pod and Deployment objects. -- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. - For most requests, minimally logging at the Metadata level is recommended - (the most basic level of logging). - -## 4.1 Worker Node Configuration Files -### 4.1.1 Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chmod 644 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) - - -**Result:** Not Applicable - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf - -### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 644, expected 644 or more restrictive OR '/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, chown root:root /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is present OR '/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig' is not present -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chmod 644 /var/lib/rancher/rke2/agent/kubelet.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'644' is equal to '644' -``` - -**Returned Value**: - -```console -permissions=644 -``` - -### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the below command (based on the file location on your system) on the each worker node. -For example, -chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig - -**Audit:** - -```bash -/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.7 Ensure that the certificate authorities file permissions are set to 644 or more restrictive (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the file permissions of the ---client-ca-file chmod 644 - -**Audit Script:** `check_cafile_permissions.sh` - -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -CAFILE=/node$CAFILE -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi - -``` - -**Audit Execution:** - -```bash -./check_cafile_permissions.sh -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) - - -**Result:** pass - -**Remediation:** -Run the following command to modify the ownership of the --client-ca-file. -chown root:root - -**Audit Script:** `check_cafile_ownership.sh` - -```bash -#!/usr/bin/env bash - -CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') -CAFILE=/node$CAFILE -if test -z $CAFILE; then CAFILE=$kubeletcafile; fi -if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi - -``` - -**Audit Execution:** - -```bash -./check_cafile_ownership.sh -``` - -**Expected Result**: - -```console -'root:root' is equal to 'root:root' -``` - -**Returned Value**: - -```console -root:root -``` - -### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 644 or more restrictive (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chmod 644 /etc/rancher/rke2/rke2.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/rancher/rke2/rke2.yaml; then stat -c permissions=%a /etc/rancher/rke2/rke2.yaml; fi' -``` - -**Expected Result**: - -```console -permissions has permissions 600, expected 644 or more restrictive -``` - -**Returned Value**: - -```console -permissions=600 -``` - -### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) - - -**Result:** pass - -**Remediation:** -Run the following command (using the config file location identified in the Audit step) -chown root:root /etc/rancher/rke2/rke2.yaml - -**Audit:** - -```bash -/bin/sh -c 'if test -e /etc/rancher/rke2/rke2.yaml; then stat -c %U:%G /etc/rancher/rke2/rke2.yaml; fi' -``` - -**Expected Result**: - -```console -'root:root' is present -``` - -**Returned Value**: - -```console -root:root -``` - -## 4.2 Kubelet -### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to -`false`. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -`--anonymous-auth=false` -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--anonymous-auth' is equal to 'false' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If -using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---authorization-mode=Webhook -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--authorization-mode' does not have 'AlwaysAllow' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to -the location of the client CA file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_AUTHZ_ARGS variable. ---client-ca-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--client-ca-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.4 Ensure that the --read-only-port argument is set to 0 (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---read-only-port=0 -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--read-only-port' is equal to '0' OR '--read-only-port' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a -value other than 0. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---streaming-connection-idle-timeout=5m -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. ---protect-kernel-defaults=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--protect-kernel-defaults' is equal to 'true' -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove the --make-iptables-util-chains argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.makeIPTablesUtilChains}' is present OR '{.makeIPTablesUtilChains}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) - - -**Result:** Not Applicable - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and remove the --hostname-override argument from the -KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -### 4.2.9 Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.eventRecordQPS}' is present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to set `tlsCertFile` to the location -of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` -to the location of the corresponding private key file. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the below parameters in KUBELET_CERTIFICATE_ARGS variable. ---tls-cert-file= ---tls-private-key-file= -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'--tls-cert-file' is present AND '--tls-private-key-file' is present -``` - -**Returned Value**: - -```console -UID PID PPID C STIME TTY TIME CMD root 3727 3667 3 23:26 ? 00:00:11 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-25-112 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=5c1cd514-db7b-4692-a1c4-cacb2656161f --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key -``` - -### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) - - -**Result:** pass - -**Remediation:** -If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or -remove it altogether to use the default value. -If using command line arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS -variable. -Based on your system, restart the kubelet service. For example, -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) - - -**Result:** pass - -**Remediation:** -Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf -on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. ---feature-gates=RotateKubeletServerCertificate=true -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{.featureGates.RotateKubeletServerCertificate}' is present OR '{.featureGates.RotateKubeletServerCertificate}' is not present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) - - -**Result:** warn - -**Remediation:** -If using a Kubelet config file, edit the file to set `TLSCipherSuites` to -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -or to a subset of these values. -If using executable arguments, edit the kubelet service file -/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and -set the --tls-cipher-suites parameter as follows, or to a subset of these values. ---tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 -Based on your system, restart the kubelet service. For example: -systemctl daemon-reload -systemctl restart kubelet.service - -**Audit:** - -```bash -/bin/ps -fC kubelet -``` - -**Audit Config:** - -```bash -/bin/cat /etc/rancher/rke2/rke2.yaml -``` - -**Expected Result**: - -```console -'{range .tlsCipherSuites[:]}{}{','}{end}' is present -``` - -**Returned Value**: - -```console -apiVersion: v1 clusters: - cluster: certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWtNU0l3SUFZRFZRUUREQmx5YTJVeUxYTmwKY25abGNpMWpZVUF4TmpjM05EVXpPVE00TUI0WERUSXpNREl5TmpJek1qVXpPRm9YRFRNek1ESXlNekl6TWpVegpPRm93SkRFaU1DQUdBMVVFQXd3WmNtdGxNaTF6WlhKMlpYSXRZMkZBTVRZM056UTFNemt6T0RCWk1CTUdCeXFHClNNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJOUHllSlcwNE9lMUN6clo5cHplN09WWlliMlh4QmNvWUJ5Z3JRcUwKYUdpR29tN2xLNGs2ZW1uaUZQekpiOU9EQ0hDRmYyVUZaVXZDQTQxcUVmNVB3NlNqUWpCQU1BNEdBMVVkRHdFQgovd1FFQXdJQ3BEQVBCZ05WSFJNQkFmOEVCVEFEQVFIL01CMEdBMVVkRGdRV0JCUnhUdGJTbTVMTnFrditkYXQrCmczalBWWXJjcERBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBN08wV1NNVHo4djgwM3BZK3hFeDR2SUZaVEMraVoKUHVDc082eFRuVVVtb2dJaEFPY1NqQVdkUVJ0UmJvaUhJbTd2RHZuM1czT1JmYlUwaXU2UUhNcXdsTGs4Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K server: https://127.0.0.1:6443 name: default contexts: - context: cluster: default user: default name: default current-context: default kind: Config preferences: {} users: - name: default user: client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrakNDQVRpZ0F3SUJBZ0lJRjc3RlZRT1l5RzR3Q2dZSUtvWkl6ajBFQXdJd0pERWlNQ0FHQTFVRUF3d1oKY210bE1pMWpiR2xsYm5RdFkyRkFNVFkzTnpRMU16a3pPREFlRncweU16QXlNall5TXpJMU16aGFGdzB5TkRBeQpNall5TXpJMU16aGFNREF4RnpBVkJnTlZCQW9URG5ONWMzUmxiVHB0WVhOMFpYSnpNUlV3RXdZRFZRUURFd3h6CmVYTjBaVzA2WVdSdGFXNHdXVEFUQmdjcWhrak9QUUlCQmdncWhrak9QUU1CQndOQ0FBUXg5Z1lxUzVCbndwTDIKcUdOeStjNnR0Yk15VmE3ZTdzT0J1UWdaWHBHZ1hNcGxOZHFhVy9lZjNZTjQwTk5RUnl2SWdWeTMzU0ZHSFJ0VgpqaXBpOXRZbG8wZ3dSakFPQmdOVkhROEJBZjhFQkFNQ0JhQXdFd1lEVlIwbEJBd3dDZ1lJS3dZQkJRVUhBd0l3Ckh3WURWUjBqQkJnd0ZvQVVyNlUvYVNMcm1ONkxiaDZHS1JJa3NoM1M1bEF3Q2dZSUtvWkl6ajBFQXdJRFNBQXcKUlFJaEFPS2paZmxRVU11RnZldlFkYzg3ckxPcnhoNUtyUGhlQUtkY0Y4YWdielFJQWlCcHBmUGNMMFRoZ1g5UAptcCtOZHJqa1hvQU5SRTlEWVdIRUlRbDdubytDdWc9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlCZVRDQ0FSK2dBd0lCQWdJQkFEQUtCZ2dxaGtqT1BRUURBakFrTVNJd0lBWURWUVFEREJseWEyVXlMV05zCmFXVnVkQzFqWVVBeE5qYzNORFV6T1RNNE1CNFhEVEl6TURJeU5qSXpNalV6T0ZvWERUTXpNREl5TXpJek1qVXoKT0Zvd0pERWlNQ0FHQTFVRUF3d1pjbXRsTWkxamJHbGxiblF0WTJGQU1UWTNOelExTXprek9EQlpNQk1HQnlxRwpTTTQ5QWdFR0NDcUdTTTQ5QXdFSEEwSUFCQlhFZ0x2Z3JLV09KdkZtVnJhNEhyY05YdmNwN3JMN3VFNW1IcFpkCmJmT2xkRDRkVlJ4NjRxak9DeUNpc2Vsczk4WDJLSXlieGNSNkpnbFU2VXRoOU5xalFqQkFNQTRHQTFVZER3RUIKL3dRRUF3SUNwREFQQmdOVkhSTUJBZjhFQlRBREFRSC9NQjBHQTFVZERnUVdCQlN2cFQ5cEl1dVkzb3R1SG9ZcApFaVN5SGRMbVVEQUtCZ2dxaGtqT1BRUURBZ05JQURCRkFpRUF3R2xqWXUxZkJpMHZROFczbWxueXVDNUJqMlBBCm14Sm1uS3BVSG8ydjJBZ0NJRndiblM3ajROUGtCT2hzRjJBeFhEZlZzdExoRWpqbmhPRHlQek1kT01STQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg== client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUpZblVvZ2U2bHVuNFN1WVVaN3VBQTVGYXV6blBaQzV2WHlpc1R2SVRjOXFvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFTWZZR0trdVFaOEtTOXFoamN2bk9yYld6TWxXdTN1N0RnYmtJR1Y2Um9GektaVFhhbWx2MwpuOTJEZU5EVFVFY3J5SUZjdDkwaFJoMGJWWTRxWXZiV0pRPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo= -``` - -## 5.1 RBAC and Service Accounts -### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) - - -**Result:** warn - -**Remediation:** -Identify all clusterrolebindings to the cluster-admin role. Check if they are used and -if they need this role or if they could use a role with fewer privileges. -Where possible, first bind users to a lower privileged role and then remove the -clusterrolebinding to the cluster-admin role : -kubectl delete clusterrolebinding [name] - -### 5.1.2 Minimize access to secrets (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove get, list and watch access to Secret objects in the cluster. - -### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) - - -**Result:** warn - -**Remediation:** -Where possible replace any use of wildcards in clusterroles and roles with specific -objects or actions. - -### 5.1.4 Minimize access to create pods (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove create access to pod objects in the cluster. - -### 5.1.5 Ensure that default service accounts are not actively used. (Automated) - - -**Result:** pass - -**Remediation:** -Create explicit service accounts wherever a Kubernetes workload requires specific access -to the Kubernetes API server. -Modify the configuration of each default service account to include this value -automountServiceAccountToken: false - -**Audit Script:** `check_for_default_sa.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) -if [[ ${count_sa} -gt 0 ]]; then - echo "false" - exit -fi - -for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") -do - for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[].kind=="ServiceAccount" and .subjects[].name=="default") or (.subjects[].kind=="Group" and .subjects[].name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') - do - read kind name <<<$(IFS=","; echo $result) - resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[] != "podsecuritypolicies")' | wc -l) - if [[ ${resource_count} -gt 0 ]]; then - echo "false" - exit - fi - done -done - - -echo "true" -``` - -**Audit Execution:** - -```bash -./check_for_default_sa.sh -``` - -**Expected Result**: - -```console -'true' is equal to 'true' -``` - -**Returned Value**: - -```console -Error from server (Forbidden): serviceaccounts is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "serviceaccounts" in API group "" at the cluster scope Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "calico-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-fleet-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-impersonation-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cattle-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "cis-operator-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "default" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-node-lease" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-public" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "local" Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "tigera-operator" true -``` - -### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) - - -**Result:** warn - -**Remediation:** -Modify the definition of pods and service accounts which do not need to mount service -account tokens to disable it. - -### 5.1.7 Avoid use of system:masters group (Manual) - - -**Result:** warn - -**Remediation:** -Remove the system:masters group from all users in the cluster. - -### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) - - -**Result:** warn - -**Remediation:** -Where possible, remove the impersonate, bind and escalate rights from subjects. - -## 5.2 Pod Security Standards -### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that either Pod Security Admission or an external policy control system is in place -for every namespace which contains user workloads. - -### 5.2.2 Minimize the admission of privileged containers (Manual) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of privileged containers. - -**Audit:** - -```bash -kubectl get psp global-restricted-psp -o json | jq -r '.spec.runAsUser.rule' -``` - -**Expected Result**: - -```console -'MustRunAsNonRoot' is present -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" -``` - -### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostPID` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostIPC` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of `hostNetwork` containers. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.7 Minimize the admission of root containers (Automated) - - -**Result:** fail - -**Remediation:** -Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` -or `MustRunAs` with the range of UIDs not including 0, is set. - -**Audit:** - -```bash -kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" --count=0 -``` - -### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) - - -**Result:** fail - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with the `NET_RAW` capability. - -**Audit:** - -```bash -kubectl get psp global-restricted-psp -o json | jq -r .spec.requiredDropCapabilities[] -``` - -**Expected Result**: - -```console -'ALL' is present -``` - -**Returned Value**: - -```console -error: the server doesn't have a resource type "psp" -``` - -### 5.2.9 Minimize the admission of containers with added capabilities (Automated) - - -**Result:** warn - -**Remediation:** -Ensure that `allowedCapabilities` is not present in policies for the cluster unless -it is set to an empty array. - -### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) - - -**Result:** warn - -**Remediation:** -Review the use of capabilites in applications running on your cluster. Where a namespace -contains applicaions which do not require any Linux capabities to operate consider adding -a PSP which forbids the admission of containers which do not drop all capabilities. - -### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. - -### 5.2.12 Minimize the admission of HostPath volumes (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers with `hostPath` volumes. - -### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) - - -**Result:** warn - -**Remediation:** -Add policies to each namespace in the cluster which has user workloads to restrict the -admission of containers which use `hostPort` sections. - -## 5.3 Network Policies and CNI -### 5.3.1 Ensure that the CNI in use supports Network Policies (Automated) - - -**Result:** pass - -**Remediation:** -If the CNI plugin in use does not support network policies, consideration should be given to -making use of a different plugin, or finding an alternate mechanism for restricting traffic -in the Kubernetes cluster. - -**Audit:** - -```bash -kubectl get pods --all-namespaces --selector='k8s-app in (calico-node, canal, cilium)' -o name | wc -l | xargs -I {} echo '--count={}' -``` - -**Expected Result**: - -```console -'count' is greater than 0 -``` - -**Returned Value**: - -```console ---count=1 -``` - -### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) - - -**Result:** true - -**Remediation:** -Follow the documentation and create NetworkPolicy objects as you need them. - -**Audit Script:** `check_for_rke2_network_policies.sh` - -```bash -#!/bin/bash - -set -eE - -handle_error() { - echo "false" -} - -trap 'handle_error' ERR - -for namespace in kube-system kube-public default; do - policy_count=$(/var/lib/rancher/rke2/bin/kubectl get networkpolicy -n ${namespace} -o json | jq -r '.items | length') - if [ ${policy_count} -eq 0 ]; then - echo "false" - exit - fi -done - -echo "true" - -``` - -**Audit Execution:** - -```bash -./check_for_rke2_network_policies.sh -``` - -**Expected Result**: - -```console -'true' is present -``` - -**Returned Value**: - -```console -true -``` - -## 5.4 Secrets Management -### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) - - -**Result:** warn - -**Remediation:** -If possible, rewrite application code to read Secrets from mounted secret files, rather than -from environment variables. - -### 5.4.2 Consider external secret storage (Manual) - - -**Result:** warn - -**Remediation:** -Refer to the Secrets management options offered by your cloud provider or a third-party -secrets management solution. - -## 5.5 Extensible Admission Control -### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and setup image provenance. - -## 5.7 General Policies -### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) - - -**Result:** warn - -**Remediation:** -Follow the documentation and create namespaces for objects in your deployment as you need -them. - -### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) - - -**Result:** warn - -**Remediation:** -Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. -An example is as below: -securityContext: -seccompProfile: -type: RuntimeDefault - -### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) - - -**Result:** warn - -**Remediation:** -Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a -suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker -Containers. - -### 5.7.4 The default namespace should not be used (Manual) - - -**Result:** warn - -**Remediation:** -Ensure that namespaces are created to allow for appropriate segregation of Kubernetes -resources and that all new resources are created in a specific namespace. diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/rancher-security-best-practices.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/rancher-security-best-practices.md new file mode 100644 index 00000000000..0d98495ffec --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/rancher-security-best-practices.md @@ -0,0 +1,21 @@ +--- +title: Rancher 安全最佳实践 +--- + + + + + +### 限制对 /version 和 /rancherversion 的公共访问 + +上游（本地） Rancher 实例提供正在运行的 Rancher 版本和用于构建它的 Go 版本信息。这些信息可以通过 `/version` 路径访问，该路径用于诸如自动化版本升级或确认部署成功等任务。上游实例还提供了可通过 `/rancherversion` 路径访问的 Rancher 版本信息。 + +攻击者可能会滥用这些信息来识别正在运行的 Rancher 版本，并与潜在的漏洞相关联以进行利用。如果你的上游 Rancher 实例在网上是公开可访问的，请使用 7 层防火墙来阻止 `/version` 和 `/rancherversion` 路径。 + +更多关于保护服务器的详细信息，请参阅 [OWASP Web Application Security Testing - Enumerate Infrastructure and Application Admin Interfaces](https://owasp.org/www-project-web-security-testing-guide/stable/4-Web_Application_Security_Testing/02-Configuration_and_Deployment_Management_Testing/05-Enumerate_Infrastructure_and_Application_Admin_Interfaces.html)。 + +### 会话管理 + +某些环境可能需要额外的安全控制来管理会话。例如，你可能希望限制用户的并发活动会话或限制可以从哪些地理位置发起这些会话。Rancher 默认情况下不支持这些功能。 + +如果你需要此类功能，请将 7 层防火墙与[外部认证](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md#外部认证与本地认证)结合使用。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/rancher-security.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/rancher-security.md new file mode 100644 index 00000000000..935aaa2b780 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/rancher-security.md @@ -0,0 +1,95 @@ +--- +title: 安全 +--- + + + + + + + + + + + +
+

安全策略

+

Rancher Labs 会负责任地披露问题，并致力于在合理的时间内解决所有问题。

+		 +

报告流程

+

请将安全问题发送至 security-rancher@suse.com。

+		 +

公告

+

订阅 Rancher 公告论坛以获取版本更新。

+
+ +安全是 Rancher 全部功能的基础。Rancher 集成了全部主流认证工具和服务，并提供了企业级的 [RBAC 功能](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md)，让你的 Kubernetes 集群更加安全。 + +本文介绍了安全相关的文档以及资源，让你的 Rancher 安装和下游 Kubernetes 集群更加安全。 + +### NeuVector 与 Rancher 的集成 + +NeuVector 是一个开源的、以容器为中心的安全应用程序，现已集成到 Rancher 中。NeuVector 提供生产安全、DevOps 漏洞保护和容器防火墙等功能。请参阅 [Rancher 文档](../../integrations-in-rancher/neuvector/neuvector.md) 和 [NeuVector 文档](https://open-docs.neuvector.com/)了解更多信息。 + +### 在 Kubernetes 集群上运行 CIS 安全扫描 + +Rancher 使用 [kube-bench](https://github.com/aquasecurity/kube-bench) 来运行安全扫描，从而检查 Kubernetes 是否按照 [CIS](https://www.cisecurity.org/cis-benchmarks/)（Center for Internet Security，互联网安全中心）Kubernetes Benchmark 中定义的安全最佳实践进行部署。 + +CIS Kubernetes Benchmark 是一个参考文档，用于为 Kubernetes 建立安全配置基线。 + +CIS 是一个 501(c\)(3) 非营利组织，成立于 2000 年 10 月，其使命是识别、开发、验证、促进和维持网络防御的最佳实践方案，并建立和指导社区，以在网络空间中营造信任的环境。 + +CIS Benchmark 是目标系统安全配置的最佳实践。CIS Benchmark 是由安全专家、技术供应商、公开和私人社区成员，以及 CIS Benchmark 开发团队共同志愿开发的。 + +Benchmark 提供两种类型的建议，分别是自动（Automated）和手动（Manual）。我们只运行 Automated 相关的测试。 + +Rancher 在集群上运行 CIS 安全扫描时会生成一份报告，该报告会显示每个测试的结果，包括测试概要以及 `passed`、`skipped` 和 `failed` 的测试数量。报告还包括失败测试的修正步骤。 + +有关详细信息，请参阅[安全扫描](../../how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md)。 + +### SELinux RPM + +[安全增强型 Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) 是对 Linux 的安全增强。被政府机构使用之后，SELinux 已成为行业标准，并在 CentOS 7 和 8 上默认启用。 + +我们提供了 `rancher-selinux` 和 `rke2-selinux` 两个 RPM（Red Hat 软件包），让 Rancher 产品能够在 SELinux 主机上正常运行。有关详细信息，请参阅[此页面](selinux-rpm/selinux-rpm.md)。 + +### Rancher 加固指南 + +Rancher 加固指南基于 CIS Kubernetes Benchmark。 + +加固指南为加固 Rancher 的生产安装提供了说明性指导。有关安全管控的完整列表，请参阅 Rancher 的 [CIS Kubernetes Benchmark 自我评估](#cis-benchmark-和自我评估)指南。 + +> 加固指南描述了如何保护集群中的节点，建议在安装 Kubernetes 之前参考加固指南中的步骤。 + +每个加固指南版本都针对特定的 CIS Kubernetes Benchmark、Kubernetes 和 Rancher 版本。 + +### CIS Benchmark 和自我评估 + +Benchmark 自我评估是 Rancher 安全加固指南的辅助。加固指南展示了如何加固集群，而 Benchmark 指南旨在帮助你评估加固集群的安全级别。 + +由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多管控验证检查都不适用。本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群的合规性。你可以前往 [CIS 网站](https://www.cisecurity.org/benchmark/kubernetes/)下载原始的 Benchmark 文档。 + +Rancher 自我评估指南的每个版本都对应于强化指南、Rancher、Kubernetes 和 CIS Benchmark 的特定版本。 + +### 第三方渗透测试报告 + +Rancher 会定期聘请第三方对 Rancher 软件栈进行安全审计和渗透测试。被测环境遵循 Rancher 在测试时提供的强化指南。以前的渗透测试报告如下。 + +结果： + +- [Cure53 渗透测试 - 2019 年 7 月](https://releases.rancher.com/documents/security/pen-tests/2019/RAN-01-cure53-report.final.pdf) +- [Untamed Theory 渗透测试 - 2019 年 3 月](https://releases.rancher.com/documents/security/pen-tests/2019/UntamedTheory-Rancher_SecurityAssessment-20190712_v5.pdf) + +请注意，新报告不再共享或公开发布。 + +### Rancher 安全公告和 CVE + +Rancher 致力于向社区通报我们产品中的安全问题。有关我们已解决的问题的 CVE（常见漏洞和暴露）列表，请参阅[此页](security-advisories-and-cves.md)。 + +### Kubernetes 安全最佳实践 + +有关保护 Kubernetes 集群的建议，请参阅 [Kubernetes 安全最佳实践](kubernetes-security-best-practices.md)指南。 + +### Rancher 安全最佳实践 + +有关保护 Rancher Manager 部署的建议，请参阅 [Rancher 安全最佳实践](rancher-security-best-practices.md)指南。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/rancher-webhook-hardening.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/rancher-webhook-hardening.md new file mode 100644 index 00000000000..e6c80bac05f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/rancher-webhook-hardening.md @@ -0,0 +1,137 @@ +--- +title: 加固 Rancher Webhook +--- + + + + + +Rancher Webhook 是 Rancher 中的一个重要组件，它在强制执行 Rancher 及其工作负载的安全要求方面发挥着重要的作用。为了减少其攻击面，对它的访问应限制为唯一有效的调用方：Kubernetes API server。这可以通过单独使用网络策略和认证，或相互结合使用以加固 webhook 免受攻击。 + +## 使用网络策略阻止外部流量 + +webhook 只应接受来自 Kubernetes API server 的请求。但是默认情况下，webhook 可以接受来自任何源的流量。如果你使用的是支持网络策略的 CNI，则可以创建一个策略来阻止来源不是 API server 的流量。 + +Kubernetes 中内置的网络策略资源无法阻止或允许来自集群主机的流量，并且 `kube-apiserver` 进程在主机网络上运行。因此你必须使用当前集群正在使用的 CNI 的高级网络策略资源。以下是 Calico 和 Cilium 的示例。更多详细的信息，请参阅你使用的 CNI 的文档。 + +### Calico + +使用 `crd.projectcalico.org/v1` API 组中的网络策略资源。使用 `app == 'rancher-webhook'` selector 为 webhook 创建一个规则，并且将 control plane 主机的 CIDR 设置为入口源： + +```yaml +apiVersion: crd.projectcalico.org/v1 +kind: NetworkPolicy +metadata: + name: allow-k8s + namespace: cattle-system +spec: + selector: app == 'rancher-webhook' + types: + - Ingress + ingress: + - action: Allow + protocol: TCP + source: + nets: + - 192.168.42.0/24 # control plane 主机的 CIDR。如果主机位于不同的子网中，则可能会列出多个。 + destination: + selector: + app == 'rancher-webhook' +``` + +### Cilium + +使用 `cilium.io/v2` API 组中的 CiliumNetworkPolicy 资源。将 `host` 和 `remote-node` 键添加到 `fromEntities` 入口规则。这会阻止集群内和外部流量，同时允许来自主机的流量。 + +```yaml +apiVersion: "cilium.io/v2" +kind: CiliumNetworkPolicy +metadata: + name: allow-k8s + namespace: cattle-system +spec: + endpointSelector: + matchLabels: + app: rancher-webhook + ingress: + - fromEntities: + - host + - remote-node +``` + +## 要求 Kubernetes API Server 对 Webhook 进行认证 + +Webhook 应仅接受来自 Kubernetes API server 的请求。默认情况下，webhook 不要求客户端对其进行认证。它可以接受任何请求。你可以配置 webhook 验证凭据，以便只有 API server 可以访问它。更多信息可以在 [Kubernetes 文档](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#authenticate-apiservers)中找到。 + +1. 配置 API server 向 webhook 提供证书，指向 AdmissionConfiguration 文件以配置 ValidatingAdmissionWebhook 和 MutatingAdmissionWebhook 插件： + + ```yaml + # /etc/rancher/admission/admission.yaml + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: ValidatingAdmissionWebhook + configuration: + apiVersion: apiserver.config.k8s.io/v1 + kind: WebhookAdmissionConfiguration + kubeConfigFile: "/etc/rancher/admission/kubeconfig" + - name: MutatingAdmissionWebhook + configuration: + apiVersion: apiserver.config.k8s.io/v1 + kind: WebhookAdmissionConfiguration + kubeConfigFile: "/etc/rancher/admission/kubeconfig" + ``` + + 这也是配置其他准入插件的同一配置文件，例如 PodSecurity。如果你的发行版或设置使用了额外的准入插件，请将它们也配置在这里。例如，将 [RKE2 的 PodSecurity 配置](https://docs.rke2.io/security/pod_security_standards)添加到这个文件中。 + +2. 创建准入插件引用的 kubeconfig 文件。Rancher Webhook 仅支持客户端证书认证，因此请生成一个 TLS 密钥对，并且将 kubeconfig 设置为使用 `client-certificate` 和 `client-key` 或者使用 `client-certificate-data` 和 `client-key-data`。 例如： + + ```yaml + # /etc/rancher/admission/kubeconfig + apiVersion: v1 + kind: Config + users: + - name: 'rancher-webhook.cattle-system.svc' + user: + client-certificate: /path/to/client/cert + client-key: /path/to/client/key + ``` + +3. 使用 `--admission-control-config-file` 参数启动 kube-apiserver 服务，并将该参数值指向你的 AdmissionConfiguration 文件。具体的操作方式因发行版而异，并不受普遍支持，例如在托管的 Kubernetes 提供商中可能不支持。请查阅你的 Kubernetes 发行版的文档以获取详细的信息。 + + 对于 RKE2，可以使用这样的配置文件启动 `rke2-server`： + + ```yaml + # /etc/rancher/rke2/config.yaml + kube-apiserver-arg: + - admission-control-config-file=/etc/rancher/admission/admission.yaml + kube-apiserver-extra-mount: + - /etc/rancher/admission:/etc/rancher/admission:ro + ``` + + :::danger + 某些发行版会默认设置此参数。如果你的发行版预配置了它自己的 AdmissionConfiguration，则必须将其包含在你自定义的准入控制的配置文件中。例如，RKE2 在 `/etc/rancher/rke2/rke2-pss.yaml` 安装了一个 AdmissionConfiguration 文件，该文件配置了 PodSecurity 准入插件。在 config.yaml 中设置 `admission-control-config-file` 将会覆盖这个重要的安全设置。要包含两个插件，请参阅 [Default Pod Security Standards 文档](https://docs.rke2.io/security/pod_security_standards)并将相应的插件配置复制到你的 admission.yaml 中。 + ::: + +4. 如果你使用 Rancher 通过现有节点来配置你的集群，请在配置集群之前在节点上创建这些文件。 + + 如果你使用 Rancher 在新节点上配置你的集群，请先允许集群配置完成，然后使用 Rancher 提供的 SSH key 和 IP 地址连接到节点，并将 RKE2 的配置文件放在 `/etc/rancher/rke2/config.yaml.d/` 目录下。 + +5. 使用这些凭证配置集群后，配置 Rancher cluster agent 以在 webhook 中启用认证。创建一个包含以下内容的 chart values 的文件： + + ```yaml + # values.yaml + auth: + clientCA: + allowedCNs: + - + - + ``` + +6. 在配置的集群中的 `cattle-system` 命名空间下创建一个 configmap，并包含这些 values 配置： + + ``` + kubectl --namespace cattle-system create configmap rancher-config --from-file=rancher-webhook=values.yaml + ``` + + webhook 将会使用这些 values 配置重新启动。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/security-advisories-and-cves.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/security-advisories-and-cves.md index c2baac88dcf..12bdf210497 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/security-advisories-and-cves.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/security-advisories-and-cves.md @@ -2,10 +2,19 @@ title: 安全公告和 CVE --- + + + + Rancher 致力于向社区披露我们产品的安全问题。我们会针对已解决的问题发布安全公告和 CVE（Common Vulnerabilities and Exposures，通用漏洞披露）。Rancher GitHub 上的[安全页面](https://github.com/rancher/rancher/security/advisories)也会发布新的安全公告。 | ID | 描述 | 日期 | 解决 | |----|-------------|------|------------| +| [CVE-2024-22030](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2024-22030) | 发现了 Rancher 和 Fleet 代理的一个漏洞，目前被认为是中到高危的 CVE。在非特定情况下，这个漏洞允许恶意行为者接管现有的 Rancher 节点。攻击者需要控制一个过期的域名，或者对该域名执行 DNS 欺骗/劫持攻击才可以利用此漏洞。被攻击的域名是 Rancher URL（用作 Rancher 集群的 server-url）。目前还没有可用的修复方案，它影响所有受支持的 Rancher 版本。建议客户和用户遵循我们[博客文章](https://www.suse.com/c/rancher-security-update/)中描述的建议和最佳实践。 | 2024 年 2 月 16 日 | 处理中 | +| [CVE-2023-32193](https://github.com/rancher/norman/security/advisories/GHSA-r8f4-hv23-6qp6) | 在 Rancher 2.6.13、2.7.9 和 2.8.1 及之前的版本中发现了一个问题。多个 Cross-Site Scripting (XSS) 漏洞可通过 Rancher UI (Norman) 进行利用。 | 2024 年 2 月 8 日 | Rancher [v2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2)、[v2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) 和 [v2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | +| [CVE-2023-32192](https://github.com/rancher/apiserver/security/advisories/GHSA-833m-37f7-jq55) | 在 Rancher 2.6.13、2.7.9 和 2.8.1 及之前的版本中发现了一个问题。多个 Cross-Site Scripting (XSS) 漏洞，可以通过 Rancher UI (Apiserver) 进行利用 | 2024 年 2 月 8 日 | Rancher [v2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2)、[v2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) 和 [v2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | +| [CVE-2023-22649](https://github.com/rancher/rancher/security/advisories/GHSA-xfj7-qf8w-2gcr) | 在 Rancher 2.6.13、2.7.9 和 2.8.1 及之前的版本中发现了一个问题。敏感数据可能会泄漏到 Rancher 的审计日志中。 | 2024 年 2 月 8 日 | Rancher [v2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2)、[v2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) 和 [v2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | +| [CVE-2023-32194](https://github.com/rancher/rancher/security/advisories/GHSA-c85r-fwc7-45vc) | 在 Rancher 2.6.13、2.7.9 和 2.8.1 及之前的版本中发现了一个问题。当为 “namespace” 资源类型授予 `create` 或 `*` 全局角色时，任何 API 组中拥有权限的用户可以管理核心 API 组中的 namespace。 | 2024 年 2 月 8 日 | Rancher [v2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2)、[v2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) 和 [v2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | | [CVE-2023-22648](https://github.com/rancher/rancher/security/advisories/GHSA-vf6j-6739-78m8) | 在 Rancher 2.6.12 和 2.7.3 及之前的版本中发现了一个问题。在用户注销并重新登录到 Rancher UI 之前，Azure AD 中的权限更改不会反映给用户。 | 2023 年 5 月 31 日 | Rancher [v2.7.4](https://github.com/rancher/rancher/releases/tag/v2.7.4) | | [CVE-2022-43760](https://github.com/rancher/rancher/security/advisories/GHSA-46v3-ggjg-qq3x) | 在 Rancher 2.6.12 和 2.7.3 及之前的版本中发现了一个问题。攻击者可以通过 Rancher UI 利用多个跨站脚本 (XSS) 漏洞。 | 2023 年 5 月 31 日 | Rancher [v2.7.4](https://github.com/rancher/rancher/releases/tag/v2.7.4) | | [CVE-2020-10676](https://github.com/rancher/rancher/security/advisories/GHSA-8vhc-hwhc-cpj4) | 在 Rancher 2.6.12 和 2.7.3 及之前的版本中发现了一个问题。具有更新命名空间权限的用户可以将该命名空间移动到他们无权访问的项目中。 | 2023 年 5 月 31 日 | Rancher [v2.7.4](https://github.com/rancher/rancher/releases/tag/v2.7.4) | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/selinux-rpm/selinux-rpm.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/selinux-rpm/selinux-rpm.md new file mode 100644 index 00000000000..2c96bf912ee --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-security/selinux-rpm/selinux-rpm.md @@ -0,0 +1,20 @@ +--- +title: SELinux RPM +--- + + + + + +[安全增强型 Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) 是对 Linux 的安全增强。 + +它由 Red Hat 开发，是 Linux 上 MAC（mandatory access controls，强制访问控制）的实现。系统管理员可以使用 MAC 设置应用程序和用户是如何访问不同资源的，例如文件、设备、网络和进程间的通信。SELinux 还通过默认限制操作系统来增强安全性。 + +被政府机构使用之后，SELinux 已成为行业标准，并在 CentOS 7 和 8 上默认启用。要检查 SELinux 是否在你的系统上启用和执行，请使用 `getenforce`： + +``` +# getenforce +Enforcing +``` + +我们提供了 [`rancher-selinux`](about-rancher-selinux.md) 和 [`rke2-selinux`](about-rke2-selinux.md) 两个 RPM（Red Hat 软件包），让 Rancher 产品能够在 SELinux 主机上正常运行。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/user-settings/user-settings.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/user-settings/user-settings.md new file mode 100644 index 00000000000..47ff9cb2518 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/user-settings/user-settings.md @@ -0,0 +1,19 @@ +--- +title: 用户设置 +--- + + + + + +在 Rancher 中，每个用户都有很多与登录相关的设置，例如个人偏好、API 密钥等。你可以从**用户设置**菜单中配置这些设置。你可以单击主菜单中的头像来打开此菜单。 + +![用户设置菜单](/img/user-settings.png) + +可用的用户设置包括： + +- [API & 密钥](api-keys.md)：如果你想以编程方式与 Rancher 交互，你需要一个 API 密钥。你可以按照本节中的说明获取密钥。 +- [云凭证](manage-cloud-credentials.md)：管理[节点模板](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md#节点模板)使用的云凭证，从而[为集群配置节点](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)。 +- [节点模板](manage-node-templates.md)：管理 [Rancher 用来为集群配置节点](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)的模板。 +- [偏好设置](user-preferences.md)：设置 Rancher UI 的表面首选项。 +- 登出：结束你的用户会话。 From 2b0a778a7d015c1297ecf8e0877e7d55be508941 Mon Sep 17 00:00:00 2001 From: Hang Jiang <43160536+jianghang8421@users.noreply.github.com> Date: Thu, 11 Apr 2024 04:50:33 +0800 Subject: [PATCH 41/67] Update missing Chinese translation for getting-started part (#1215) --- .../air-gapped-upgrades.md | 32 +- ...install-upgrade-on-a-kubernetes-cluster.md | 351 ++++++++++++++++++ .../rancher-on-aks.md | 8 +- .../rancher-on-amazon-eks.md | 6 +- .../rancher-on-gke.md | 6 +- .../rollbacks.md | 50 ++- .../upgrades.md | 6 +- .../installation-and-upgrade.md | 94 +++++ .../installation-references/feature-flags.md | 4 +- .../helm-chart-options.md | 8 +- .../installation-references.md | 5 + .../install-docker.md | 14 +- .../installation-requirements.md | 219 +++++++++++ .../port-requirements.md | 10 +- .../air-gapped-helm-cli-install.md | 31 ++ .../infrastructure-private-registry.md | 4 +- .../publish-images.md | 4 +- .../other-installation-methods.md | 19 + .../install-rancher.md | 2 + .../rancher-behind-an-http-proxy.md | 13 + .../set-up-infrastructure.md | 2 +- .../certificate-troubleshooting.md | 2 + .../rancher-on-a-single-node-with-docker.md | 209 +++++++++++ .../roll-back-docker-installed-rancher.md | 4 +- .../upgrade-docker-installed-rancher.md | 2 +- .../resources/add-tls-secrets.md | 2 +- .../resources/choose-a-rancher-version.md | 2 +- .../resources/helm-version-requirements.md | 4 +- .../resources/local-system-charts.md | 2 +- .../resources/resources.md | 25 ++ .../resources/update-rancher-certificate.md | 2 +- .../resources/upgrade-cert-manager.md | 6 +- .../upgrade-and-roll-back-kubernetes.md | 4 +- ...de-kubernetes-without-upgrading-rancher.md | 2 +- .../current/getting-started/overview.md | 20 +- .../deploy-rancher-manager/aws-marketplace.md | 6 +- .../deploy-rancher-manager/aws.md | 4 +- .../deploy-rancher-manager/azure.md | 4 +- .../deploy-rancher-manager.md | 20 + .../deploy-rancher-manager/digitalocean.md | 4 +- .../deploy-rancher-manager/equinix-metal.md | 6 +- .../deploy-rancher-manager/gcp.md | 4 +- .../deploy-rancher-manager/helm-cli.md | 6 +- .../deploy-rancher-manager/hetzner-cloud.md | 4 +- .../deploy-rancher-manager/linode.md | 79 ++++ .../deploy-rancher-manager/outscale-qs.md | 4 +- .../deploy-rancher-manager/vagrant.md | 4 +- .../deploy-workloads/deploy-workloads.md | 8 + .../quick-start-guides/quick-start-guides.md | 17 + .../air-gapped-upgrades.md | 32 +- ...install-upgrade-on-a-kubernetes-cluster.md | 351 ++++++++++++++++++ .../rancher-on-aks.md | 8 +- .../rancher-on-amazon-eks.md | 6 +- .../rancher-on-gke.md | 6 +- .../upgrades.md | 6 +- .../installation-and-upgrade.md | 94 +++++ .../installation-references/feature-flags.md | 4 +- .../helm-chart-options.md | 8 +- .../installation-references.md | 5 + .../install-docker.md | 14 +- .../installation-requirements.md | 215 +++++++++++ .../port-requirements.md | 10 +- .../air-gapped-helm-cli-install.md | 31 ++ .../infrastructure-private-registry.md | 4 +- .../install-rancher-ha.md | 6 +- .../publish-images.md | 8 +- .../other-installation-methods.md | 19 + .../install-rancher.md | 6 +- .../rancher-behind-an-http-proxy.md | 13 + .../set-up-infrastructure.md | 2 +- .../certificate-troubleshooting.md | 2 + .../rancher-on-a-single-node-with-docker.md | 209 +++++++++++ .../roll-back-docker-installed-rancher.md | 4 +- .../upgrade-docker-installed-rancher.md | 2 +- .../resources/add-tls-secrets.md | 2 +- .../resources/choose-a-rancher-version.md | 2 +- .../resources/helm-version-requirements.md | 4 +- .../resources/local-system-charts.md | 2 +- .../resources/resources.md | 25 ++ .../resources/update-rancher-certificate.md | 2 +- .../resources/upgrade-cert-manager.md | 9 +- .../upgrade-and-roll-back-kubernetes.md | 4 +- ...de-kubernetes-without-upgrading-rancher.md | 2 +- .../version-2.8/getting-started/overview.md | 20 +- .../deploy-rancher-manager/aws-marketplace.md | 6 +- .../deploy-rancher-manager/aws.md | 4 +- .../deploy-rancher-manager/azure.md | 4 +- .../deploy-rancher-manager.md | 20 + .../deploy-rancher-manager/digitalocean.md | 4 +- .../deploy-rancher-manager/equinix-metal.md | 6 +- .../deploy-rancher-manager/gcp.md | 4 +- .../deploy-rancher-manager/helm-cli.md | 14 +- .../deploy-rancher-manager/hetzner-cloud.md | 4 +- .../deploy-rancher-manager/linode.md | 79 ++++ .../deploy-rancher-manager/outscale-qs.md | 4 +- .../deploy-rancher-manager/vagrant.md | 4 +- .../deploy-workloads/deploy-workloads.md | 8 + .../quick-start-guides/quick-start-guides.md | 17 + 98 files changed, 2453 insertions(+), 206 deletions(-) create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-and-upgrade.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/installation-references.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/other-installation-methods.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/resources.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/linode.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/quick-start-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-and-upgrade.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/installation-references.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/other-installation-methods.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/resources.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/linode.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/quick-start-guides.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades.md index dee4c863bcf..8e5c0eca63f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades.md @@ -1,5 +1,5 @@ --- -title: 在离线环境中渲染 Helm 模板 +title: 在离线环境中升级 --- :::note @@ -35,6 +35,26 @@ helm template rancher ./rancher-.tgz --output-dir . \ --set useBundledSystemChart=true # Use the packaged Rancher system charts ``` +#### 解决 UPGRADE FAILED 错误 + +如果你遇到错误消息 `Error: UPGRADE FAILED: "rancher" has no deployed releases`，Rancher 可能是通过 `helm template` 命令安装的。要成功升级 Rancher，请改用以下命令： + +``` +helm template rancher ./rancher-.tgz --output-dir . \ + --no-hooks \ # prevent files for Helm hooks from being generated + --namespace cattle-system \ + --set hostname= \ + --set certmanager.version= \ + --set rancherImage=/rancher/rancher \ + --set systemDefaultRegistry= \ # Set a default private registry to be used in Rancher + --set useBundledSystemChart=true # Use the packaged Rancher system charts +``` + +执行 Helm 命令后，需要应用渲染后的模板： + +``` +kubectl -n cattle-system apply -R -f ./rancher +``` ### 选项 B：使用 Kubernetes 密文从文件中获取证书 ```plain @@ -62,16 +82,6 @@ helm template rancher ./rancher-.tgz --output-dir . \ --set useBundledSystemChart=true # Use the packaged Rancher system charts ``` -### 应用已渲染的模板 - -将渲染的 manifest 目录复制到可以访问 Rancher Server 集群的系统中，并应用渲染的模板。 - -使用 `kubectl` 来应用渲染的 manifest。 - -```plain -kubectl -n cattle-system apply -R -f ./rancher -``` - ## 验证升级 登录 Rancher 以确认升级成功。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md new file mode 100644 index 00000000000..437bc35815e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md @@ -0,0 +1,351 @@ +--- +title: 在 Kubernetes 集群上安装/升级 Rancher +description: 了解如何在开发和生产环境中安装 Rancher。了解单节点和高可用安装 +--- + +在本节中，你将学习如何使用 Helm CLI 在 Kubernetes 集群上部署 Rancher。 + + +## 先决条件 + +- [Kubernetes 集群](#kubernetes-集群) +- [Ingress Controller](#ingress-controller) +- [CLI 工具](#cli-工具) + +### Kubernetes 集群 + +设置 Rancher Server 的本地 Kubernetes 集群。 + +Rancher 可以安装在任何 Kubernetes 集群上。这个集群可以使用上游 Kubernetes，也可以使用 Rancher 的 Kubernetes 发行版之一，也可以是来自 Amazon EKS 等提供商的托管 Kubernetes 集群。 + +你可参考以下教程，以获得设置 Kubernetes 集群的帮助： + +- **RKE**：[安装 RKE Kubernetes 集群的教程](../../../how-to-guides/new-user-guides/kubernetes-cluster-setup/rke1-for-rancher.md)；[为高可用 RKE 集群设置基础设施的教程](../../../how-to-guides/new-user-guides/infrastructure-setup/ha-rke1-kubernetes-cluster.md)。 +- **K3s**：[安装 K3s Kubernetes 集群的教程](../../../how-to-guides/new-user-guides/kubernetes-cluster-setup/k3s-for-rancher.md)；[设置高可用 K3s 集群的基础设施的教程](../../../how-to-guides/new-user-guides/infrastructure-setup/ha-k3s-kubernetes-cluster.md)。 +- **RKE2:** ：[安装 RKE2 Kubernetes 集群的教程](../../../how-to-guides/new-user-guides/kubernetes-cluster-setup/rke2-for-rancher.md)；[设置高可用 RKE2 集群的基础设施的教程](../../../how-to-guides/new-user-guides/infrastructure-setup/ha-rke2-kubernetes-cluster.md)。 +- **Amazon EKS**：[在 Amazon EKS 上安装 Rancher 以及如何安装 Ingress Controller 以访问 Rancher Server](rancher-on-amazon-eks.md)。 +- **AKS**：[使用 Azure Kubernetes 服务安装 Rancher 以及如何安装 Ingress Controller 以访问 Rancher Server](rancher-on-aks.md)。 +- **GKE**：有关如何使用 GKE 安装 Rancher，包括如何安装 Ingress Controller 以便可以访问 Rancher Server，请参阅[此页面](rancher-on-gke.md)。GKE 创建 Kubernetes 集群时有两种运行模式，分别是 Autopilot 和 Standard 模式。Autopilot 模式的集群配置对编辑 kube-system 命名空间有限制。但是，Rancher 在安装时需要在 kube-system 命名空间中创建资源。因此，你将无法在以 Autopilot 模式创建的 GKE 集群上安装 Rancher。 + + +### Ingress Controller + +Rancher UI 和 API 通过 Ingress 公开。换言之，安装 Rancher 的 Kubernetes 集群必须包含一个 Ingress Controller。 + +对于 RKE、RKE2 和 K3s，你不需要手动安装 Ingress Controller，因为它是默认安装的。 + +对于默认不包含 Ingress Controller 的发行版（例如 EKS、GKE 或 AKS 等托管 Kubernetes 集群），你必须先部署 Ingress Controller。请注意，Rancher Helm Chart 默认情况下不会在 Ingress 上设置 `ingressClassName`。因此，你必须将 Ingress Controller 配置为在没有 `ingressClassName` 的情况下也可以监视 Ingress。 + +上面的 **Amazon EKS**、**AKS** 和 **GKE** 教程中包含了示例。 + +### CLI 工具 + +设置 Kubernetes 集群需要以下 CLI 工具。请确保这些工具已安装并在你的 `$PATH` 中可用。 + +- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl) - Kubernetes 命令行工具。 +- [Helm](https://docs.helm.sh/using_helm/#installing-helm) - Kubernetes 的包管理器。请参见 [Helm 版本要求](../resources/helm-version-requirements.md)选择 Helm 版本来安装 Rancher。请为你的具体平台参见 [Helm 项目提供的说明](https://helm.sh/docs/intro/install/)。 + +## 安装 Rancher Helm Chart + +Rancher 是使用 Kubernetes 的 [Helm](https://helm.sh/) 包管理器安装的。Helm Chart 为 Kubernetes YAML 清单文件提供了模板语法。通过 Helm，用户可以创建可配置的 deployment，而不仅仅只能使用静态文件。 + +如果系统无法直接访问互联网，请参见[离线环境：Kubernetes 安装](../other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha.md)。 + +如果要指定安装的 Rancher 版本，请参见[选择 Rancher 版本](../resources/choose-a-rancher-version.md)。 + +如果要指定用于安装 Rancher 的 Helm 版本，请参见[Helm 版本要求](../resources/helm-version-requirements.md)。 + +:::note + +本安装指南假定你使用的是 Helm 3。 + +::: + +要设置 Rancher： + +1. [添加 Helm Chart 仓库](#1-添加-helm-chart-仓库) +2. [为 Rancher 创建命名空间](#2-为-rancher-创建命名空间) +3. [选择 SSL 配置](#3-选择-ssl-配置) +4. [安装 cert-manager](#4-安装-cert-manager)（除非你自带证书，否则 TLS 将在负载均衡器上终止） +5. [使用 Helm 和你选择的证书选项安装 Rancher](#5-根据你选择的证书选项通过-helm-安装-rancher) +6. [验证 Rancher Server 是否部署成功](#6-验证-rancher-server-是否部署成功) +7. [保存选项](#7-保存选项) + +### 1. 添加 Helm Chart 仓库 + +执行 `helm repo add` 命令，以添加包含安装 Rancher 的 Chart 的 Helm Chart 仓库。有关如何选择仓库，以及哪个仓库最适合你的用例，请参见[选择 Rancher 版本](../getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md)。 + +- Latest：建议用于试用最新功能 + ``` + helm repo add rancher-latest https://releases.rancher.com/server-charts/latest + ``` +- Stable：建议用于生产环境 + ``` + helm repo add rancher-stable https://releases.rancher.com/server-charts/stable + ``` +- Alpha：即将发布的实验性预览。 + ``` + helm repo add rancher-alpha https://releases.rancher.com/server-charts/alpha + ``` + 注意：不支持升级到 Alpha 版、从 Alpha 版升级或在 Alpha 版之间升级。 + +### 2. 为 Rancher 创建命名空间 + +你需要定义一个 Kubernetes 命名空间，用于安装由 Chart 创建的资源。这个命名空间的名称为 `cattle-system`： + +``` +kubectl create namespace cattle-system +``` + +### 3. 选择 SSL 配置 + +Rancher Management Server 默认需要 SSL/TLS 配置来保证访问的安全性。 + +:::note + +如果你想在外部终止 SSL/TLS，请参见[外部负载均衡器的 TLS 终止](../getting-started/installation-and-upgrade/installation-references/helm-chart-options.md#外部-tls-终止)。 + +::: + +你可以从以下三种证书来源中选择一种，用于在 Rancher Server 中终止 TLS： + +- **Rancher 生成的 TLS 证书**：要求你在集群中安装 `cert-manager`。Rancher 使用 `cert-manager` 签发并维护证书。Rancher 会生成自己的 CA 证书，并使用该 CA 签署证书。然后 `cert-manager`负责管理该证书。 +- **Let's Encrypt**：Let's Encrypt 选项也需要使用 `cert-manager`。但是，在这种情况下，cert-manager 与 Let's Encrypt 的特殊颁发者相结合，该颁发者执行获取 Let's Encrypt 颁发的证书所需的所有操作（包括请求和验证）。此配置使用 HTTP 验证（`HTTP-01`），因此负载均衡器必须具有可以从互联网访问的公共 DNS 记录。 +- **你已有的证书**：使用已有的 CA 颁发的公有或私有证书。Rancher 将使用该证书来保护 WebSocket 和 HTTPS 流量。在这种情况下，你必须上传名称分别为 `tls.crt` 和 `tls.key`的 PEM 格式的证书以及相关的密钥。如果你使用私有 CA，则还必须上传该 CA 证书。这是由于你的节点可能不信任此私有 CA。Rancher 将获取该 CA 证书，并从中生成一个校验和，各种 Rancher 组件将使用该校验和来验证其与 Rancher 的连接。 + + +| 配置 | Helm Chart 选项 | 是否需要 cert-manager | +| ------------------------------ | ----------------------- | ------------------------------------- | +| Rancher 生成的证书（默认） | `ingress.tls.source=rancher` | [是](#4-安装-cert-manager) | +| Let’s Encrypt | `ingress.tls.source=letsEncrypt` | [是](#4-安装-cert-manager) | +| 你已有的证书 | `ingress.tls.source=secret` | 否 | + +### 4. 安装 cert-manager + +> 如果你使用自己的证书文件（`ingress.tls.source=secret`）或使用[外部负载均衡器的 TLS 终止](../installation-references/helm-chart-options.md#外部-tls-终止)，你可以跳过此步骤。 + +仅在使用 Rancher 生成的证书（`ingress.tls.source=rancher`）或 Let's Encrypt 颁发的证书（`ingress.tls.source=letsEncrypt`）时，才需要安装 cert-manager。 + +
+ 单击展开 + +:::note 重要提示： + +由于 cert-manager 的最新改动，你需要升级 cert-manager 版本。如果你需要升级 Rancher 并使用低于 0.11.0 的 cert-manager 版本，请参见[升级文档](../resources/upgrade-cert-manager.md)。 + +::: + +这些说明来自 [cert-manager 官方文档](https://cert-manager.io/docs/installation/kubernetes/#installing-with-helm)。 + +:::note + +要查看自定义 cert-manager 安装的选项（包括集群使用 PodSecurityPolicies 的情况），请参阅 [cert-manager 文档](https://artifacthub.io/packages/helm/cert-manager/cert-manager#configuration)。 + +::: + +``` +# 如果你手动安装了CRD，而不是在 Helm 安装命令中添加了 `--set installCRDs=true` 选项，你应该在升级 Helm Chart 之前升级 CRD 资源。 +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download//cert-manager.crds.yaml + +# 添加 Jetstack Helm 仓库 +helm repo add jetstack https://charts.jetstack.io + +# 更新本地 Helm Chart 仓库缓存 +helm repo update + +# 安装 cert-manager Helm Chart +helm install cert-manager jetstack/cert-manager \ + --namespace cert-manager \ + --create-namespace +``` + +安装完 cert-manager 后，你可以通过检查 cert-manager 命名空间中正在运行的 Pod 来验证它是否已正确部署： + +``` +kubectl get pods --namespace cert-manager + +NAME READY STATUS RESTARTS AGE +cert-manager-5c6866597-zw7kh 1/1 Running 0 2m +cert-manager-cainjector-577f6d9fd7-tr77l 1/1 Running 0 2m +cert-manager-webhook-787858fcdb-nlzsq 1/1 Running 0 2m +``` + +
+ +### 5. 根据你选择的证书选项，通过 Helm 安装 Rancher + +不同的证书配置需要使用不同的 Rancher 安装命令。 + +但是，无论证书如何配置，Rancher 在 `cattle-system` 命名空间中的安装名称应该总是 `rancher`。 + +:::tip 测试和开发： + +这个安装 Rancher 的最终命令需要一个将流量转发到 Rancher 的域名。如果你使用 Helm CLI 设置概念证明，则可以在传入 `hostname` 选项时使用伪域名。伪域名的一个例子是 `.sslip.io`，这会把 Rancher 暴露在它运行的 IP 上。生产安装中要求填写真实的域名。 + +::: + + + + +默认情况是使用 Rancher 生成 CA，并使用 `cert-manager` 颁发用于访问 Rancher Server 接口的证书。 + +由于 `rancher` 是 `ingress.tls.source` 的默认选项，因此在执行 `helm install` 命令时，我们不需要指定 `ingress.tls.source`。 + +- 将 `hostname` 设置为解析到你的负载均衡器的 DNS 名称。 +- 将 `bootstrapPassword` 设置为 `admin` 用户独有的值。 +- 如果你需要安装指定的 Rancher 版本，使用 `--version` 标志，例如 `--version 2.7.0`。 +- 对于 Kubernetes v1.25 或更高版本，使用 Rancher v2.7.2-v2.7.4 时，将 `global.cattle.psp.enabled` 设置为 `false`。对于 Rancher v2.7.5 及更高版本来说，这不是必需的，但你仍然可以手动设置该选项。 + +``` +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set bootstrapPassword=admin +``` + +如果你安装的是 alpha 版本，Helm 会要求你在安装命令中添加 `--devel` 选项： + +``` +helm install rancher rancher-alpha/rancher --devel +``` + +等待 Rancher 运行： + +``` +kubectl -n cattle-system rollout status deploy/rancher +Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... +deployment "rancher" successfully rolled out +``` + + + + +此选项使用 `cert-manager` 来自动请求和续订 [Let's Encrypt](https://letsencrypt.org/) 证书。Let's Encrypt 是免费的，而且是受信的 CA，因此可以为你提供有效的证书。 + +:::note + +由于 HTTP-01 质询只能在端口 80 上完成，因此你需要打开端口 80。 + +::: + +在以下命令中， + +- 将 `hostname` 设置为公有 DNS 记录。 +- 将 `bootstrapPassword` 设置为 `admin` 用户独有的值。 +- 将 `ingress.tls.source` 设置为 `letsEncrypt`。 +- 将 `letsEncrypt.email` 设置为可通讯的电子邮件地址，用于发送通知（例如证书到期的通知）。 +- 将 `letsEncrypt.ingress.class` 设为你的 Ingress Controller（例如 `traefik`，`nginx`，`haproxy`） +- 对于 Kubernetes v1.25 或更高版本，使用 Rancher v2.7.2-v2.7.4 时，将 `global.cattle.psp.enabled` 设置为 `false`。对于 Rancher v2.7.5 及更高版本来说，这不是必需的，但你仍然可以手动设置该选项。 + +``` +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set bootstrapPassword=admin \ + --set ingress.tls.source=letsEncrypt \ + --set letsEncrypt.email=me@example.org \ + --set letsEncrypt.ingress.class=nginx +``` + +如果你安装的是 alpha 版本，Helm 会要求你在安装命令中添加 `--devel` 选项： + +``` +helm install rancher rancher-alpha/rancher --devel +``` + +等待 Rancher 运行： + +``` +kubectl -n cattle-system rollout status deploy/rancher +Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... +deployment "rancher" successfully rolled out +``` + + + +在此选项中，你使用你自己的证书来创建 Kubernetes 密文，以供 Rancher 使用。 + +运行这个命令时，`hostname` 选项必须与服务器证书中的 `Common Name` 或 `Subject Alternative Names` 条目匹配，否则 Ingress controller 将无法正确配置。 + +虽然技术上仅需要 `Subject Alternative Names` 中有一个条目，但是拥有一个匹配的 `Common Name` 可以最大程度地提高与旧版浏览器/应用的兼容性。 + +:::note + +如果你想检查证书是否正确，请查看[如何在服务器证书中检查 Common Name 和 Subject Alternative Names](../../../faq/technical-items.md#如何在服务器证书中检查-common-name-和-subject-alternative-names)。 + +::: + +- 设置 `hostname`。 +- 将 `bootstrapPassword` 设置为 `admin` 用户独有的值。 +- 将 `ingress.tls.source` 设置为 `secret`。 +- 对于 Kubernetes v1.25 或更高版本，使用 Rancher v2.7.2-v2.7.4 时，将 `global.cattle.psp.enabled` 设置为 `false`。对于 Rancher v2.7.5 及更高版本来说，这不是必需的，但你仍然可以手动设置该选项。 + +``` +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set bootstrapPassword=admin \ + --set ingress.tls.source=secret +``` +如果你安装的是 alpha 版本，Helm 会要求你在安装命令中添加 `--devel` 选项： + +``` +helm install rancher rancher-alpha/rancher --devel +``` + +如果你使用的是私有 CA 证书，请在命令中增加 `--set privateCA=true`。 + +``` +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set bootstrapPassword=admin \ + --set ingress.tls.source=secret \ + --set privateCA=true +``` + +**添加 TLS 密文（千万不要遗漏此步骤）**：现在 Rancher 已经完成部署，你还需要参考[添加 TLS 密文](../resources/add-tls-secrets.md)发布证书文件，以便 Rancher 和 Ingress Controller 可以使用它们。 + + + + +Rancher Chart 有许多选项，用于为你的具体环境自定义安装。以下是一些常见的高级方案： + +- [HTTP 代理](../installation-references/helm-chart-options.md#http-代理) +- [私有容器镜像仓库](../installation-references/helm-chart-options.md#私有仓库和离线安装) +- [外部负载均衡器上的 TLS 终止](../installation-references/helm-chart-options.md#外部-tls-终止) + +如需获取完整的选项列表，请参见 [Chart 选项](../installation-references/helm-chart-options.md)。 + + +### 6. 验证 Rancher Server 是否部署成功 + +添加密文后，检查 Rancher 是否已成功运行： + +``` +kubectl -n cattle-system rollout status deploy/rancher +Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... +deployment "rancher" successfully rolled out +``` + +如果你看到 `error: deployment "rancher" exceeded its progress deadline` 这个错误，可运行以下命令来检查 deployment 的状态： + +``` +kubectl -n cattle-system get deploy rancher +NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE +rancher 3 3 3 3 3m +``` + +`DESIRED` 和 `AVAILABLE`的个数应该相同。 + +### 7. 保存选项 + +请保存你使用的 `--set` 选项。使用 Helm 升级 Rancher 到新版本时，你将需要使用相同的选项。 + +### 安装完成 + +安装已完成。现在 Rancher Server 应该已经可以正常运行了。 + +使用浏览器打开把流量转发到你的负载均衡器的 DNS 域名。然后，你就会看到一个漂亮的登录页面了。 + +如果遇到任何问题，请参见[故障排除](troubleshooting.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md index 22de9e7af35..4db511d97a0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md @@ -6,7 +6,7 @@ title: 在 Azure Kubernetes Service 上安装 Rancher 本指南使用命令行工具来配置一个带有 Ingress 的 AKS 集群。如果你更喜欢使用 Azure 门户来配置集群，请参见[官方文档](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-portal)。 -如果你已有一个 AKS Kubernetes 集群，请直接跳到[安装 Ingress](#5-安装-ingress) 的步骤，然后按照[此页](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。 +如果你已有一个 AKS Kubernetes 集群，请直接跳到[安装 Ingress](#5-安装-ingress) 的步骤，然后按照[此页](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。 ## 先决条件 @@ -19,7 +19,7 @@ title: 在 Azure Kubernetes Service 上安装 Rancher - [Microsoft Azure 账号](https://azure.microsoft.com/en-us/free/)：用于创建部署 Rancher 和 Kubernetes 的资源。 - [Microsoft Azure 订阅](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription#create-a-subscription-in-the-azure-portal)：如果你没有的话，请访问此链接查看如何创建 Microsoft Azure 订阅。 - [Micsoroft Azure 租户](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-create-new-tenant)：访问此链接并参考教程以创建 Microsoft Azure 租户。 -- 你的订阅有足够的配额，至少有 2 个 vCPU。有关 Rancher Server 资源要求的详情，请参见[此节](../../../pages-for-subheaders/installation-requirements.md#rke-和托管-kubernetes)。 +- 你的订阅有足够的配额，至少有 2 个 vCPU。有关 Rancher Server 资源要求的详情，请参见[此节](../installation-requirements/installation-requirements.md)。 - 在 Azure 中用 Helm 安装 Rancher 时，请使用 L7 负载均衡器来避免网络问题。详情请参见 [Azure 负载均衡器限制](https://docs.microsoft.com/en-us/azure/load-balancer/components#limitations)。 ## 1. 准备你的工作站 @@ -134,7 +134,7 @@ ingress-nginx-controller LoadBalancer 10.0.116.18 40.31.180.83 80:31229 ## 8. 安装 Rancher Helm Chart -按照[本页](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。任何 Kubernetes 发行版上安装的 Rancher 的 Helm 说明都是一样的。 +按照[本页](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。任何 Kubernetes 发行版上安装的 Rancher 的 Helm 说明都是一样的。 安装 Rancher 时，使用上一步获取的 DNS 名称作为 Rancher Server 的 URL。它可以作为 Helm 选项传递进来。例如，如果 DNS 名称是 `rancher.my.org`，你需要使用 `--set hostname=rancher.my.org` 选项来运行 Helm 安装命令。 @@ -144,4 +144,4 @@ ingress-nginx-controller LoadBalancer 10.0.116.18 40.31.180.83 80:31229 --set ingress.ingressClassName=nginx ``` -请参阅[Helm 安装命令](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 +请参阅[Helm 安装命令](install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md index 335e3fee425..efc95fe7e13 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md @@ -4,7 +4,7 @@ title: 在 Amazon EKS 上安装 Rancher 本文介绍了如何在 Amazon EKS 集群上安装 Rancher。你也可以[通过 AWS Marketplace 安装 Rancher](../../quick-start-guides/deploy-rancher-manager/aws-marketplace.md)。 -如果你已经有一个 EKS Kubernetes 集群，请直接跳转到[安装 Ingress](#5-安装-ingress)这个步骤。然后按照[此处](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的步骤安装 Rancher Helm Chart。 +如果你已经有一个 EKS Kubernetes 集群，请直接跳转到[安装 Ingress](#5-安装-ingress)这个步骤。然后按照[此处](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的步骤安装 Rancher Helm Chart。 ## 为 Rancher Server 创建 EKS 集群 @@ -138,7 +138,7 @@ ingress-nginx-controller LoadBalancer 10.100.90.18 a904a952c73bf4f668a17c4 ### 8. 安装 Rancher Helm Chart -按照[本页](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。任何 Kubernetes 发行版上安装的 Rancher 的 Helm 说明都是一样的。 +按照[本页](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。任何 Kubernetes 发行版上安装的 Rancher 的 Helm 说明都是一样的。 安装 Rancher 时，使用上一步获取的 DNS 名称作为 Rancher Server 的 URL。它可以作为 Helm 选项传递进来。例如，如果 DNS 名称是 `rancher.my.org`，你需要使用 `--set hostname=rancher.my.org` 选项来运行 Helm 安装命令。 @@ -148,4 +148,4 @@ ingress-nginx-controller LoadBalancer 10.100.90.18 a904a952c73bf4f668a17c4 --set ingress.ingressClassName=nginx ``` -请参阅[Helm 安装命令](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 +请参阅[Helm 安装命令](install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md index 90d395c0f12..04c2d4ff181 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md @@ -4,13 +4,13 @@ title: 在 GKE 集群上安装 Rancher 在本节中，你将学习如何使用 GKE 安装 Rancher。 -如果你已经有一个 GKE Kubernetes 集群，请直接跳转到[安装 Ingress](#7-安装-ingress)这个步骤。然后按照[此处](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的步骤安装 Rancher Helm Chart。 +如果你已经有一个 GKE Kubernetes 集群，请直接跳转到[安装 Ingress](#7-安装-ingress)这个步骤。然后按照[此处](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的步骤安装 Rancher Helm Chart。 ## 先决条件 - 你需要有一个 Google 账号。 - 你需要有一个 Google Cloud Billing 账号。你可使用 Google Cloud Console 来管理你的 Cloud Billing 账号。有关 Cloud Console 的详情，请参见 [ Console 通用指南](https://support.google.com/cloud/answer/3465889?hl=en&ref_topic=3340599)。 -- 你需要至少一个在用的 IP 地址和至少 2 个 CPU 的云配额。有关 Rancher Server 的硬件要求，请参见[本节](../../../pages-for-subheaders/installation-requirements.md#rke-和托管-kubernetes)。 +- 你需要至少一个在用的 IP 地址和至少 2 个 CPU 的云配额。有关 Rancher Server 的硬件要求，请参见[本节](../installation-requirements/installation-requirements.md)。 ## 1. 启用 Kubernetes Engine API @@ -190,7 +190,7 @@ ingress-nginx-controller LoadBalancer 10.3.244.156 35.233.206.34 80:3187 --set ingress.ingressClassName=nginx ``` -请参阅[Helm 安装命令](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 +请参阅[Helm 安装命令](install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 在 Rancher v2.7.5 中，如果你打算在集群上使用默认的 GKE Ingress 而不启用 VPC 原生的集群模式，则需要设置以下标志： diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md index c306729f752..d5c661c421f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md @@ -2,16 +2,37 @@ title: 回滚 --- -## 使用 Rancher 2.6.4+ 进行回滚的其他步骤 +本页概述了如何在升级 Rancher 后将 Rancher 回滚到之前的版本。 + +请在以下情况时按照本页的说明进行操作： +- 正在运行的 Rancher 实例在备份完成后升级到了更新的版本。 +- 上游（本地）集群与进行备份的集群相同。 + +:::tip + +* [请参阅这些步骤来迁移 Rancher](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md). +* 如果你需要还原 Rancher 到同一版本的之前的状态, 请参阅[还原 Rancher](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md). + +::: + +## 特殊场景下的操作 + +在以下场景中需要执行替代步骤来进行回滚： +- 从 v2.6.4 及更高版本回滚到 v2.6.x 的早期版本。 +- 从 v2.7.7 及更高版本回滚到 v2.7.x 的早期版本。 Rancher v2.6.4 将 cluster-api 模块从 v0.4.4 升级到 v1.0.2。反过来，cluster-api 的 v1.0.2 版本将集群 API 的自定义资源定义 (CRD) 从 `cluster.x-k8s.io/v1alpha4` 升级到 `cluster.x-k8s.io/v1beta1`。当你尝试将 Rancher v2.6.4 回滚到以前版本的 Rancher v2.6.x 时，CRD 升级到 v1beta1 会导致回滚失败。这是因为使用旧 apiVersion (v1alpha4) 的 CRD 与 v1beta1 不兼容。 -要避免回滚失败，你需要在尝试恢复操作或回滚**之前**运行以下 Rancher 脚本： +在 Rancher v2.7.7 版本中，应用 `rancher-provisioning-capi` 会自动安装在上游（本地）集群上，来替代嵌入的 cluster-api 控制器。 如果上游集群同时包含该应用和 Rancher v2.7.6 及更早版本，则会发生冲突和意外错误。 因此，如果你尝试从 Rancher v2.7.7 回滚到 Rancher v2.7.x 的任何早期版本，需要执行替代的步骤。 + +### 步骤 1： 清理上游（本地）集群 + +要避免回滚失败，你需要在尝试恢复操作或回滚**之前**根据该[说明](https://github.com/rancher/rancher-cleanup/blob/main/README.md)运行以下 Rancher 脚本： + -* `verify.sh`：检查集群中是否有任何与 Rancher 相关的资源。 * `cleanup.sh`：清理集群。 +* `verify.sh`：检查集群中是否有任何与 Rancher 相关的资源。 -有关详细信息和源代码，请参阅 [rancher/rancher-cleanup repo](https://github.com/rancher/rancher-cleanup)。 :::caution @@ -19,12 +40,17 @@ Rancher v2.6.4 将 cluster-api 模块从 v0.4.4 升级到 v1.0.2。反过来，c ::: -### 从 v2.6.4+ 回滚到较低版本的 v2.6.x +**结果：** 上游（本地）集群中所有 Rancher 关联的资源将被清理。 -1. 按照[说明](https://github.com/rancher/rancher-cleanup/blob/main/README.md)运行脚本。 -1. 按照[说明](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)在现有集群上安装 rancher-backup Helm Chart 并恢复之前的状态。 - 1. 省略步骤 3。 - 1. 执行到步骤 4 时，在要回滚到的 local 集群上安装 Rancher 2.6.x 版本。 +请参阅 [rancher/rancher-cleanup 仓库](https://github.com/rancher/rancher-cleanup) 以获取更多细节和源码。 + +### 步骤 2： 还原备份并启动 Rancher + +此时上游集群上应该已经没有 Rancher 相关的资源了。 因此，下一步与将 Rancher 迁移到不包含 Rancher 资源的新集群相同。 + +按照[说明](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)安装 Rancher-Backup Helm Chart 并恢复 Rancher 到之前的状态。请记住： +1. 步骤 3 可以跳过，因为之前安装的 Cert-Manager 应用应该还在上游（本地）集群中。 +2. 执行到步骤 4 时，在要回滚到的 local 集群上安装你需要的 Rancher 版本。 ## 回滚到 Rancher 2.5.0+ @@ -42,14 +68,14 @@ Rancher v2.6.4 将 cluster-api 模块从 v0.4.4 升级到 v1.0.2。反过来，c ::: -### 创建 Restore 自定义资源 +### 步骤 1 ：创建 Restore 自定义资源 1. 点击 **☰ > 集群管理**。 1. 找到你的本地集群，并点击 **Explore**。 1. 在左侧导航栏中，点击 **Rancher 备份 > 还原**。 :::note - 如果 Rancher Backups 应用不可见，你需要到 **Apps** 的 Charts 页面中安装应用。详情请参见[此处](../../../pages-for-subheaders/helm-charts-in-rancher.md#charts)。 + 如果 Rancher Backups 应用不可见，你需要到 **Apps** 的 Charts 页面中安装应用。详情请参见[此处](../../../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md#access-charts)。 ::: @@ -91,7 +117,7 @@ kubectl get pods -n cattle-resources-system kubectl logs -n cattle-resources-system -f ``` -### 回滚到上一个 Rancher 版本 +### 步骤 2：回滚到上一个 Rancher 版本 你可以使用 Helm CLI 回滚 Rancher。要回滚到上一个版本： diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades.md index 9e5367abb2d..b60d3ab748b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades.md @@ -28,11 +28,13 @@ kubeconfig 也可以通过 `--kubeconfig` 标签（详情请参见 https://helm. 本安装指南假定你使用的是 Helm 3。 + + 如果你使用 Helm 2，请参见 [Helm 2 迁移到 Helm 3 文档](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/)。如果你不能升级到 Helm 3，[Helm 2 升级页面](/versioned_docs/version-2.0-2.4/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades/helm2.md)提供了使用 Helm 2 升级的旧升级指南。 ### 离线安装：推送镜像到私有镜像仓库 -[仅适用于离线安装](../../../pages-for-subheaders/air-gapped-helm-cli-install.md)：为新的 Rancher Server 版本收集和推送镜像。使用你需要针对 Rancher 版本升级的镜像，按照步骤[推送镜像到私有镜像仓库](../other-installation-methods/air-gapped-helm-cli-install/publish-images.md)。 +[仅适用于离线安装](../other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)：为新的 Rancher Server 版本收集和推送镜像。使用你需要针对 Rancher 版本升级的镜像，按照步骤[推送镜像到私有镜像仓库](../other-installation-methods/air-gapped-helm-cli-install/publish-images.md)。 ### 使用 cert-manager 0.8.0 之前的版本升级 @@ -59,7 +61,7 @@ kubeconfig 也可以通过 `--kubeconfig` 标签（详情请参见 https://helm. 1. 获取你用来安装 Rancher 的仓库名称。 - 关于仓库及其区别，请参见 [Helm Chart Repositories](../resources/choose-a-rancher-version.md#helm-chart-仓库)。 + 关于仓库及其区别，请参见 [Helm Chart 仓库](../resources/choose-a-rancher-version.md#helm-chart-仓库)。 - Latest：建议用于试用最新功能 ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-and-upgrade.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-and-upgrade.md new file mode 100644 index 00000000000..aeef9f9c6e7 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-and-upgrade.md @@ -0,0 +1,94 @@ +--- +title: 安装/升级 Rancher +description: 了解如何在开发和生产环境中安装 Rancher。了解单节点和高可用安装 +--- + +本节介绍了 Rancher 各种安装方式以及每个安装方式的优点。 + +## 名词解释 + +本章节涉及以下名词： + +- **Rancher Server**：用于管理和配置 Kubernetes 集群。你可以通过 Rancher Server 的 UI 与下游 Kubernetes 集群进行交互。Rancher Management Server 可以安装到任意 Kubernetes 集群上，包括托管的集群，如 Amazon EKS 集群。 +- **RKE（Rancher Kubernetes Engine）**：是经过认证的 Kubernetes 发行版，也是用于创建和管理 Kubernetes 集群的 CLI 工具和库。 +- **K3s（轻量级 Kubernetes）**：也是经过认证的 Kubernetes 发行版。它比 RKE 更新，更易用且更轻量，其所有组件都在一个小于 100 MB 的二进制文件中。 +- **RKE2**：一个完全合规的 Kubernetes 发行版，专注于安全和合规性。 + +`restrictedAdmin` Helm Chart 选项在 **Rancher Server** 可用。如果该选项设置为 true，初始的 Rancher 用户访问本地 Kubernetes 集群会受到限制，以避免权限升级。详情请参见 [restricted-admin 角色](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md#受限管理员)。 + +## 安装方式概述 + +Rancher 可以安装在以下主要架构上： + +### 使用 Helm CLI 安装的高可用 Kubernetes + +我们建议使用 Kubernetes 包管理器 Helm 在专用的 Kubernetes 集群上安装 Rancher。在 RKE 集群中，需要使用三个节点才能实现高可用集群。在 K3s 集群中，只需要两个节点即可。 + +### 通过 AWS Marketplace 在 EKS 上安装 Rancher + +你可以[通过 AWS Marketplace](../quick-start-guides/deploy-rancher-manager/aws-marketplace.md) 将 Rancher 安装到 Amazon Elastic Kubernetes Service (EKS) 上。部署的 EKS 集群已生产就绪，并遵循 AWS 最佳实践。 + +### 单节点 Kubernetes 安装 + +Rancher 可以安装在单节点 Kubernetes 集群上。但是，在单节点安装的情况下，Rancher Server 没有高可用性。而高可用性对在生产环境中运行 Rancher 非常重要。 + +但是，如果你想要短期内使用单节点节省资源，同时又保留高可用性迁移路径，那么单节点 Kubernetes 安装也是合适的。你也可以之后向集群中添加节点，获得高可用的 Rancher Server。 + +### Docker 安装 + +如果你的目的是测试或演示，你可以使用 Docker 把 Rancher 安装到单个节点中。本地 Kubernetes 集群是安装到单个 Docker 容器中的，而 Rancher 是安装到本地集群中的。 + +Rancher backup operator 可将 Rancher 从单个 Docker 容器迁移到高可用 Kubernetes 集群上。详情请参见[把 Rancher 迁移到新集群](../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)。 + +### 其他方式 + +如果你需要在离线环境中或使用 HTTP 代理安装 Rancher，请参见以下独立的说明文档： + +| 网络访问方式 | 基于 Kubernetes 安装（推荐） | 基于 Docker 安装 | +| ---------------------------------- | ------------------------------ | ---------- | +| 可直接访问互联网 | [文档](install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md) | [文档](other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md) | +| 使用 HTTP 代理 | [文档](other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md) | [文档](other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md)及[配置](../../reference-guides/single-node-rancher-in-docker/http-proxy-configuration.md) | +| 离线环境 | [文档](other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md) | [文档](other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md) | + +我们建议在 Kubernetes 集群上安装 Rancher，因为在多节点集群中，Rancher Server 可以实现高可用。高可用配置可以提升 Rancher 访问其管理的下游 Kubernetes 集群的稳定性。 + +因此，我们建议在生产级别的架构中，设置一个高可用的 Kubernetes 集群，然后在这个集群上安装 Rancher。安装 Rancher 后，你可以使用 Rancher 部署和管理 Kubernetes 集群。 + +如果你的目的是测试或演示，你可以将 Rancher 安装到单个 Docker 容器中。Docker 安装可以让你实现开箱即用，以使用 Rancher 设置 Kubernetes 集群。Docker 安装主要是用于探索 Rancher Server 的功能，只适用于开发和测试。 + +[在 Kubernetes 上安装 Rancher 的说明](install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)介绍了如何首先使用 K3s 或 RKE 创建和管理 Kubernetes 集群，然后再将 Rancher 安装到该集群上。 + +如果 Kubernetes 集群中的节点正在运行且满足[节点要求](installation-requirements/installation-requirements.md)，你可以使用 Helm 将 Rancher 部署到 Kubernetes 上。Helm 使用 Rancher 的 Helm Chart 在 Kubernetes 集群的每个节点上安装 Rancher 的副本。我们建议使用负载均衡器将流量定向到集群中的每个 Rancher 副本上。 + +如需进一步了解 Rancher 架构，请参见[架构概述](../../reference-guides/rancher-manager-architecture/rancher-manager-architecture.md)，[生产级别架构推荐](../../reference-guides/rancher-manager-architecture/architecture-recommendations.md)或[最佳实践指南](../../reference-guides/best-practices/rancher-server/tips-for-running-rancher.md)。 + +## 先决条件 + +安装 Rancher 之前，请确保你的节点满足所有[安装要求](installation-requirements/installation-requirements.md)。 + +## 架构建议 + +为了达到最佳性能和安全性，我们建议你为 Rancher Management Server 使用单独的专用 Kubernetes 集群。不建议在此集群上运行用户工作负载。部署 Rancher 后，你可以[创建或导入集群](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)来运行你的工作负载。 + +详情请参见[架构推荐](../../reference-guides/rancher-manager-architecture/architecture-recommendations.md)。 + +### 在 Kubernetes 上安装 Rancher 的更多选项 + +参见 [Helm Chart 选项](installation-references/helm-chart-options.md)以了解在 Kubernetes 集群上安装 Rancher 的其他配置，包括： + +- [开启 API 审计日志来记录所有事务](installation-references/helm-chart-options.md#api-审计日志) +- [负载均衡器上的 TLS 终止](installation-references/helm-chart-options.md#外部-tls-终止) +- [自定义 Ingress](installation-references/helm-chart-options.md#自定义-ingress) + +在 Rancher 的安装指南中，我们推荐使用 K3s 或 RKE 来配置 Kubernetes 集群，然后再在这个集群中安装 Rancher。K3s 和 RKE 均提供许多配置选项，用于为你的具体环境自定义 Kubernetes 集群。有关选项和功能的完整列表，请参见： + +- [RKE 配置选项](https://rancher.com/docs/rke/latest/en/config-options/) +- [K3s 配置选项](https://rancher.com/docs/k3s/latest/en/installation/install-options/) + +### 在 Docker 上安装 Rancher 的更多选项 + +参见 [Docker 安装选项](other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md)了解其他配置，包括： + +- [开启 API 审计日志来记录所有事务](../../reference-guides/single-node-rancher-in-docker/advanced-options.md#api-审计日志) +- [外部负载均衡器](../../how-to-guides/advanced-user-guides/configure-layer-7-nginx-load-balancer.md) +- [持久化数据存储](../../reference-guides/single-node-rancher-in-docker/advanced-options.md#持久化数据) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/feature-flags.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/feature-flags.md index 873a0689b14..5a022b13135 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/feature-flags.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/feature-flags.md @@ -4,7 +4,7 @@ title: 功能开关 使用功能开关（Feature Flag），你可以试用可选或实验性的功能并启用正在逐步淘汰的旧版功能。 -要了解功能的值以及如何启用它们，请参阅[启用实验性功能](../../../pages-for-subheaders/enable-experimental-features.md)。 +要了解功能的值以及如何启用它们，请参阅[启用实验性功能](../../../how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 :::note @@ -16,7 +16,7 @@ title: 功能开关 - `continuous-delivery`：允许从 Fleet 中单独禁用 Fleet GitOps。有关详细信息，请参阅[持续交付](../../../how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md)。 - `fleet`：v2.6 及更高版本的 Rancher 配置框架需要 Fleet。即使你在旧 Rancher 版本中禁用了该标志，该标志也将在升级时自动启用。有关详细信息，请参阅 [Fleet - GitOps at Scale](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md)。 -- `harvester`：管理 Virtualization Management 页面的访问。用户可以在该页面直接导航到 Harvester 集群并访问 Harvester UI。有关详细信息，请参阅 [Harvester 集成](../../../integrations-in-rancher/harvester.md)。 +- `harvester`：管理 Virtualization Management 页面的访问。用户可以在该页面直接导航到 Harvester 集群并访问 Harvester UI。有关详细信息，请参阅 [Harvester 集成](../../../integrations-in-rancher/harvester/overview.md)。 - `istio-virtual-service-ui`：启用[可视界面](../../../how-to-guides/advanced-user-guides/enable-experimental-features/istio-traffic-management-features.md)来创建、读取、更新和删除 Istio 虚拟服务和目标规则，这些都是 Istio 流量管理功能。 - `legacy`：启用 2.5.x 及更早版本的一组功能，这些功能正逐渐被新的实现淘汰。它们是已弃用以及后续可用于新版本的功能组合。新的 Rancher 安装会默认禁用此标志。如果你从以前版本的 Rancher 升级，此标志会启用。 - `multi-cluster-management`：允许配置和管理多个 Kubernetes 集群。此标志只能在安装时设置。后续无法启用或禁用它。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index bd69a89cfee..c924cc3c797 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -7,7 +7,7 @@ keywords: [rancher helm chart, rancher helm 选项, rancher helm chart 选项, h 如需选择 Helm Chart 版本，请参见[本页](../../../getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md)。 -了解开启实验性功能的详情，请参见[本页](../../../pages-for-subheaders/enable-experimental-features.md)。 +了解开启实验性功能的详情，请参见[本页](../../../how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 ## 常用选项 @@ -81,13 +81,13 @@ kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ 启用 [API 审计日志](../../../how-to-guides/advanced-user-guides/enable-api-audit-log.md)。 -你可以像收集其他容器日志一样收集此日志。在 Rancher Server 集群上为 `System` 项目启用 [Logging](../../../pages-for-subheaders/logging.md)。 +你可以像收集其他容器日志一样收集此日志。在 Rancher Server 集群上为 `System` 项目启用 [Logging](../../../integrations-in-rancher/logging/logging.md)。 ```plain --set auditLog.level=1 ``` -默认情况下，启用审计日志会在 Rancher pod 中创建一个 Sidecar 容器。这个容器（`rancher-audit-log`）会把日志流传输到 `stdout`。你可以像收集其他容器日志一样收集此日志。如果你使用 Sidecar 作为审计日志的目标时， `hostPath`，`maxAge`，`maxBackups` 和 `maxSize` 选项不会生效。建议使用你的操作系统或 Docker Daemon 的日志轮换功能来控制磁盘空间的使用。请为 Rancher Server 集群或 System 项目启用 [Logging](../../../pages-for-subheaders/logging.md)。 +默认情况下，启用审计日志会在 Rancher pod 中创建一个 Sidecar 容器。这个容器（`rancher-audit-log`）会把日志流传输到 `stdout`。你可以像收集其他容器日志一样收集此日志。如果你使用 Sidecar 作为审计日志的目标时， `hostPath`，`maxAge`，`maxBackups` 和 `maxSize` 选项不会生效。建议使用你的操作系统或 Docker Daemon 的日志轮换功能来控制磁盘空间的使用。请为 Rancher Server 集群或 System 项目启用 [Logging](../../../integrations-in-rancher/logging/logging.md)。 将 `auditLog.destination` 的值设为 `hostPath`，可以将日志转发到与主机系统共享的卷，而不是传输到 Sidecar 容器。如果目标设置为 `hostPath`，你可能需要调整其他 auditLog 参数以进行日志轮换。 @@ -202,7 +202,7 @@ kubectl -n cattle-system create secret generic tls-ca-additional --from-file=ca- ### 私有仓库和离线安装 -有关使用私有仓库安装 Rancher 的详情，请参见[离线安装](../../../pages-for-subheaders/air-gapped-helm-cli-install.md)。 +有关使用私有仓库安装 Rancher 的详情，请参见[离线安装](../other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)。 ## 外部 TLS 终止 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/installation-references.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/installation-references.md new file mode 100644 index 00000000000..c2d9035c3cd --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/installation-references.md @@ -0,0 +1,5 @@ +--- +title: 安装参考 +--- + +有关其他安装资源，请参阅以下参考指南：[Rancher Helm Chart 选项](helm-chart-options.md)、[TLS 设置](tls-settings.md)和[功能开关](feature-flags.md)。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-requirements/install-docker.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-requirements/install-docker.md index 8b98ce09030..27bf949bb11 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-requirements/install-docker.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-requirements/install-docker.md @@ -2,22 +2,22 @@ title: 安装 Docker --- -在使用 Helm 或 Docker 在 RKE 集群节点上安装 Rancher Server 前，你需要先安装 Docker。RKE2 和 K3s 集群不要求使用 Docker。 +在使用 Helm 在 RKE 集群节点上或使用 Docker 安装 Rancher Server 前，你需要在节点中先安装 Docker。RKE2 和 K3s 集群不要求使用 Docker。 Docker 有几个安装方法。一种方法是参见 [Docker 官方文档](https://docs.docker.com/install/)以了解如何在 Linux 上安装 Docker。不同 Linux 发行版的安装步骤可能有所不同。 -另一种方式是使用 Rancher 的 Docker 安装脚本，该脚本可用于较新的 Docker 版本。 +另一种方式是使用 Rancher 的 Docker 安装脚本，该脚本可用于较新的 Docker 版本。 Rancher 为每个 Kubernetes 支持的上游 Docker 版本提供了安装脚本。 -例如，此命令可用于在 SUSE Linux Enterprise 或 Ubuntu 等主要 Linux 发行版上安装 Docker 20.10： +例如，此命令可用于在 SUSE Linux Enterprise 或 Ubuntu 等主要 Linux 发行版上安装 Docker ： -``` -curl https://releases.rancher.com/install-docker/20.10.sh | sh +```bash +curl https://releases.rancher.com/install-docker/.sh | sh ``` -Rancher 提供 Kubernetes 支持的所有上游 Docker 版本的安装脚本。如需了解我们是否提供某个 Docker 版本的安装脚本，请参见包含了 Rancher 所有的 Docker 安装脚本的 [GitHub 仓库](https://github.com/rancher/install-docker)。 +请参阅 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix)，使用匹配你的操作系统和 Rancher 版本并且经过验证的 Docker 版本。 尽管支持矩阵列出了经过验证的 Docker 版本直至补丁版本，但只有发行版的主要版本和次要版本与 Docker 安装脚本相关。 请注意，必须应用以下 sysctl 设置： -``` +```bash net.bridge.bridge-nf-call-iptables=1 ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md new file mode 100644 index 00000000000..e4800c2528f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md @@ -0,0 +1,219 @@ +--- +title: 安装要求 +description: 如果 Rancher 配置在 Docker 或 Kubernetes 中运行时，了解运行 Rancher Server 的每个节点的节点要求 +--- + +本文描述了对需要安装 Rancher Server 的节点的软件、硬件和网络要求。Rancher Server 可以安装在单个节点或高可用的 Kubernetes 集群上。 + +:::note 重要提示： + +如果你需要在 Kubernetes 集群上安装 Rancher，该节点的要求与用于运行应用和服务的[下游集群的节点要求](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)不同。 + +::: + +Rancher UI 在基于 Firefox 或 Chromium 的浏览器（Chrome、Edge、Opera、Brave）中效果最佳。 + +查看我们的[最佳实践](../../../reference-guides/best-practices/rancher-server/tips-for-running-rancher.md)页面，获取在生产环境中运行 Rancher Server 的建议。 + +## Kubernetes 与 Rancher 的兼容性 + +Rancher 需要安装在支持的 Kubernetes 版本上。请查阅 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions)，确保你的 Kubernetes 版本受支持。 + +## 在安全加固的 Kubernetes 集群上安装 Rancher + +如果你在安全加固的 Kubernetes 集群上安装 Rancher，请查看[豁免必须的 Rancher 命名空间](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md#豁免必须的-rancher-命名空间)以了解详细的要求。 + +## 操作系统和容器运行时要求 + +所有支持的操作系统都使用 64-bit x86 架构。Rancher 兼容当前所有的主流 Linux 发行版。 + +[Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions)列出了每个 Rancher 版本测试过的操作系统和 Docker 版本。 + +运行 RKE 集群的节点需要安装 Docker。RKE2 或 K3s 集群不需要它。 + +请安装 `ntp`（Network Time Protocol），以防止在客户端和服务器之间由于时间不同步造成的证书验证错误。 + +某些 Linux 发行版的默认防火墙规则可能会阻止 Kubernetes 集群内的通信。从 Kubernetes v1.19 开始，你必须关闭 firewalld，因为它与 Kubernetes 网络插件冲突。 + +如果你不太想这样做的话，你可以查看[相关问题](https://github.com/rancher/rancher/issues/28840)中的建议。某些用户已能成功[使用 ACCEPT 策略 为 Pod CIDR 创建一个独立的 firewalld 区域](https://github.com/rancher/rancher/issues/28840#issuecomment-787404822)。 + +如果你需要在 ARM64 上使用 Rancher，请参见[在 ARM64（实验功能）上运行 Rancher](../../../how-to-guides/advanced-user-guides/enable-experimental-features/rancher-on-arm64.md)。 + +### RKE2 要求 + +对于容器运行时，RKE2 附带了自己的 containerd。RKE2 安装不需要 Docker。 + +如需了解 RKE2 通过了哪些操作系统版本的测试，请参见 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions)。 + + +### K3s 要求 + +对于容器运行时，K3s 默认附带了自己的 containerd。你也可以将 K3s 配置为使用已安装的 Docker 运行时。有关在 Docker 中使用 K3s 的更多信息，请参阅 [K3s 文档](https://docs.k3s.io/advanced#using-docker-as-the-container-runtime)。 + +Rancher 需要安装在支持的 Kubernetes 版本上。如需了解你使用的 Rancher 版本支持哪些 Kubernetes 版本，请参见 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions)。如需指定 K3s 版本，在运行 K3s 安装脚本时，使用 `INSTALL_K3S_VERSION` 环境变量。 + +如果你使用 **Raspbian Buster** 在 K3s 集群上安装 Rancher，请按照[这些步骤](https://rancher.com/docs/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster)切换到旧版 iptables。 + +如果你使用 Alpine Linux 的 K3s 集群上安装 Rancher，请按照[这些步骤](https://rancher.com/docs/k3s/latest/en/advanced/#additional-preparation-for-alpine-linux-setup)进行其他设置。 + +### RKE 要求 + +RKE 需要 Docker 容器运行时。支持的 Docker 版本请参见 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions) + +有关详细信息，请参阅[安装 Docker](install-docker.md)。 + +## 硬件要求 + +本节描述安装 Rancher Server 的节点的 CPU、内存和磁盘要求。硬件要求根据你的 Rancher 部署规模而定。 + +### 实际考虑 + +Rancher 的硬件占用空间取决于许多因素，包括： + + - 托管的基础设施规模 (例如： 节点数量，集群数量)。 + - 所需访问控制规则的复杂性（例如：RoleBinding 对象计数）。 + - 工作负载数量 (例如： Kubernetes 部署，Fleet 部署)。 + - 使用模式 (例如：主动使用的功能集合，使用频率，并发用户数量). + +由于存在许多可能随时间变化的影响因素，因此此处列出的要求为适合大多数用例的起点。 然而，你的用例可能有不同的要求。 若你需要对于特定场景的咨询，请[联系 Rancher]((https://rancher.com/contact/)) 以获得进一步指导。 + + +特别指出，本页面中的要求基于以下假设的环境提出，包括： + - 每种类型的 Kubernetes 资源数量小于 60,000 个。 + - 每个节点最多 120 个 Pod。 + - 上游（本地）集群中最多 200 个 CRD。 + - 下游集群中最多 100 个 CRD。 + - 最多 50 个 Fleet 部署。 + +更多的数量也是能够达到的，但需要更高的硬件要求。 如果你有超过 20,000 个相同类型的资源，通过 Rancher UI 加载整个列表的时间可能需要几秒钟。 + +:::note Evolution: + +Rancher 的代码库不断发展，用例不断变化，Rancher 积累的经验也在不断增长。 + +随着指导方针的准确性不断的提高并且变得更加具体，硬件要求也会发生变化。 + +如果你发现你的 Rancher 部署不再符合列出的建议，请[联系 Rancher](https://rancher.com/contact/) 进行重新评估。 + +::: + +### RKE2 Kubernetes + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，生产环境下的高可用安装最少需要 3 个节点。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | +| --------------- | -------- | --------- | ----- | ---- | +| 小 | 150 | 1500 | 4 | 16 GB | +| 中 | 300 | 3000 | 8 | 32 GB | +| 大 (*) | 500 | 5000 | 16 | 64 GB | +| 更大 (†) | (†) | (†) | (†) | (†) | + +(*)： 大规模的部署需要你[遵循最佳实践](../../../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md)以获得足够的性能。 + +(†)： 通过特别的硬件建议和调整能够实现更大的部署规模。 你可以[联系 Rancher](https://rancher.com/contact/) 进行定制评估。 + +有关 RKE2 一般要求的更多详细信息，请参见 [RKE2 文档](https://docs.rke2.io/install/requirements)。 + +### K3s Kubernetes + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，生产环境下的高可用安装最少需要 3 个节点。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | 外部数据库(*) | +| --------------- | ---------- | ------------ | -------| ---------| ------------------------- | +| Small | 150 | 1500 | 4 | 16 GB | 2 vCPUs, 8 GB + 1000 IOPS | +| Medium | 300 | 3000 | 8 | 32 GB | 4 vCPUs, 16 GB + 2000 IOPS | +| Large (†) | 500 | 5000 | 16 | 64 GB | 8 vCPUs, 32 GB + 4000 IOPS | + +(*)：外部数据库是指将 K3s 集群数据存储在[专用的外部主机](https://docs.k3s.io/datastore)上。 这是可选的。 具体要求取决于使用的外部数据库。 + +(†)：大规模的部署需要你[遵循最佳实践](../../../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md)以获得足够的性能。 + +有关 K3s 一般要求的更多详细信息，请参见 [K3s 文档](https://docs.k3s.io/installation/requirements)。 + +### 托管 Kubernetes + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，生产环境下的高可用安装最少需要 3 个节点。 + +这些要求适用于托管 Kubernetes 集群，例如 Amazon Elastic Kubernetes Service (EKS)、Azure Kubernetes Service (AKS) 或 Google Kubernetes Engine (GKE)。 它们不适用于 Rancher SaaS 解决方案，例如 [Rancher Prime Hosted](https://www.rancher.com/products/rancher)。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | +|-----------------------------|----------------------------|-------------------------|-------|-------| +| 小 | 150 | 1500 | 4 | 16 GB | +| 中 | 300 | 3000 | 8 | 32 GB | +| 大 (*) | 500 | 5000 | 16 | 64 GB | + +(*)：大规模的部署需要你[遵循最佳实践](../../../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md)以获得足够的性能。 + + +### RKE + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，生产环境下的高可用安装最少需要 3 个节点。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | +|-----------------------------|----------------------------|-------------------------|-------|-------| +| 小 | 150 | 1500 | 4 | 16 GB | +| 中 | 300 | 3000 | 8 | 32 GB | +| 大 (*) | 500 | 5000 | 16 | 64 GB | + +(*)： 大规模的部署需要你[遵循最佳实践](../../../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md)以获得足够的性能。 + +有关 RKE 一般要求的更多详细信息，请参见 [RKE 文档](https://rke.docs.rancher.com/os)。 + +### Docker + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，在 Docker 中安装 Rancher 仅适用于开发或测试目的。不建议在生产环境中使用。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | +|-----------------------------|----------------------------|-------------------------|-------|------| +| 小 | 5 | 50 | 1 | 4 GB | +| 中 | 15 | 200 | 2 | 8 GB | + +## Ingress + +安装 Rancher 的 Kubernetes 集群中的每个节点都应该运行一个 Ingress。 + +Ingress 需要部署为 DaemonSet 以确保负载均衡器能成功把流量转发到各个节点。 + +如果是 RKE，RKE2 和 K3s 安装，你不需要手动安装 Ingress，因为它是默认安装的。 + +对于托管的 Kubernetes 集群（EKS、GKE、AKS），你需要设置 Ingress。 + +- **Amazon EKS**：[在 Amazon EKS 上安装 Rancher 以及如何安装 Ingress 以访问 Rancher Server](../install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md)。 +- **AKS**：[使用 Azure Kubernetes 服务安装 Rancher 以及如何安装 Ingress 以访问 Rancher Server](../install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md)。 +- **GKE**：[使用 GKE 安装 Rancher 以及如何安装 Ingress 以访问 Rancher Server](../install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md)。 + +## 磁盘 + +etcd 在集群中的性能决定了 Rancher 的性能。因此，为了获得最佳速度，我们建议使用 SSD 磁盘来支持 Rancher 管理的 Kubernetes 集群。在云提供商上，你还需使用能获得最大 IOPS 的最小大小。在较大的集群中，请考虑使用专用存储设备存储 etcd 数据和 wal 目录。 + +## 网络要求 + +本节描述了安装 Rancher Server 的节点的网络要求。 + +:::caution + +如果包含 Rancher 的服务器带有 `X-Frame-Options=DENY` 标头，在升级旧版 UI 之后，Rancher UI 中的某些页面可能无法渲染。这是因为某些旧版页面在新 UI 中是以 iFrames 模式嵌入的。 + +::: + +### 节点 IP 地址 + +无论你是在单个节点还是高可用集群上安装 Rancher，每个节点都应配置一个静态 IP。如果使用 DHCP，则每个节点都应该有一个 DHCP 预留，以确保节点分配到相同的 IP 地址。 + +### 端口要求 + +为了确保能正常运行，Rancher 需要在 Rancher 节点和下游 Kubernetes 集群节点上开放一些端口。不同集群类型的 Rancher 和下游集群的所有必要端口，请参见[端口要求](port-requirements.md)。 + +## Dockershim 支持 + +有关 Dockershim 支持的详情，请参见[此页面](dockershim.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-requirements/port-requirements.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-requirements/port-requirements.md index 7dcdc48683c..0bf72d2e721 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-requirements/port-requirements.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-requirements/port-requirements.md @@ -178,9 +178,9 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点 下游 Kubernetes 集群用于运行你的应用和服务。本节介绍了哪些端口需要在下游集群的节点上打开，以便 Rancher 能够与它们进行通信。 -不同的下游集群的启动方式有不同的端口要求。下面的每个标签都列出了不同[集群类型](../../../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)所需打开的端口。 +不同的下游集群的启动方式有不同的端口要求。下面的每个标签都列出了不同[集群类型](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)所需打开的端口。 -下图描述了为每个[集群类型](../../../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)打开的端口。 +下图描述了为每个[集群类型](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)打开的端口。
Rancher 管理面板的端口要求
@@ -202,7 +202,7 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点
单击展开 -下表描述了节点在[云提供商](../../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md)中创建的情况下，[Rancher 启动 Kubernetes](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md) 的端口要求。 +下表描述了节点在[云提供商](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)中创建的情况下，[Rancher 启动 Kubernetes](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) 的端口要求。 :::note @@ -219,7 +219,7 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点
单击展开 -下表描述了使用[自定义节点](../../../pages-for-subheaders/use-existing-nodes.md)的情况下，[Rancher 启动 Kubernetes](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md) 的端口要求。 +下表描述了使用[自定义节点](../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)的情况下，[Rancher 启动 Kubernetes](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) 的端口要求。 @@ -230,7 +230,7 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点
单击展开 -下表描述了[托管集群](../../../pages-for-subheaders/set-up-clusters-from-hosted-kubernetes-providers.md)的端口要求。 +下表描述了[托管集群](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md)的端口要求。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md new file mode 100644 index 00000000000..4e36f893125 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md @@ -0,0 +1,31 @@ +--- +title: 离线 Helm CLI 安装 +--- + +本文介绍如何使用 Helm CLI 在离线环境中安装 Rancher Server。离线环境可以是 Rancher Server 离线安装、防火墙后面或代理后面。 + +Rancher 安装在 RKE Kubernetes 集群、K3s Kubernetes 集群，或单个 Docker 容器上对应的安装步骤会有所不同。 + +如需了解各个安装方式的更多信息，请参见[本页](../../installation-and-upgrade.md)。 + +在安装指导中，我们为不同的安装选项提供对应的 _选项卡_ 。 + +:::note 重要提示： + +如果你按照 Docker 安装指南安装 Rancher，你将没有把 Docker 安装转换为 Kubernetes 安装的升级途径。 + +::: + +## 安装概要 + +1. [设置基础设施和私有镜像仓库](infrastructure-private-registry.md) +2. [收集镜像到私有镜像仓库](publish-images.md) +3. [设置 Kubernetes 集群（如果你使用 Docker 安装，请跳过此步骤）](install-kubernetes.md) +4. [安装 Rancher](install-rancher-ha.md) + +## 升级 + +如需在离线环境中使用 Helm CLI 升级 Rancher，请按照[升级步骤](../../install-upgrade-on-a-kubernetes-cluster/upgrades.md)进行操作。 + +### 后续操作 +[准备节点](infrastructure-private-registry.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/infrastructure-private-registry.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/infrastructure-private-registry.md index e3e7be1a627..8978a639b0e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/infrastructure-private-registry.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/infrastructure-private-registry.md @@ -6,7 +6,7 @@ title: '1. 设置基础设施和私有镜像仓库' 离线环境是 Rancher Server 离线安装或安装在防火墙后面的环境。 -Rancher 安装在 K3s Kubernetes 集群、RKE Kubernetes 集群还是单个 Docker 容器上对应的基础设施设置会有所不同。如需了解各个安装方式的更多信息，请参见[本页](../../../../pages-for-subheaders/installation-and-upgrade.md)。 +Rancher 安装在 K3s Kubernetes 集群、RKE Kubernetes 集群还是单个 Docker 容器上对应的基础设施设置会有所不同。如需了解各个安装方式的更多信息，请参见[本页](../../installation-and-upgrade.md)。 Rancher 可以安装在任何 Kubernetes 集群上。为了阅读方便，我们在下文中仍提供了 RKE 和 K3s Kubernetes 基础设施教程。 @@ -25,7 +25,7 @@ Rancher 可以安装在任何 Kubernetes 集群上。为了阅读方便，我们 这些主机会断开互联网链接，但需要能与你的私有镜像仓库连接。 -请确保你的节点满足[操作系统，容器运行时，硬件和网络](../../../../pages-for-subheaders/installation-requirements.md)的常规要求。 +请确保你的节点满足[操作系统，容器运行时，硬件和网络](../../installation-requirements/installation-requirements.md)的常规要求。 如需获取配置 Linux 节点的示例，请参见[在 Amazon EC2 中配置节点](../../../../how-to-guides/new-user-guides/infrastructure-setup/nodes-in-amazon-ec2.md)的教程。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md index e3240faf9c4..783555ba158 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md @@ -4,11 +4,11 @@ title: '2. 收集镜像并发布到私有仓库' 本文介绍如何配置私有镜像仓库，以便在安装 Rancher 时，Rancher 可以从此私有镜像仓库中拉取所需的镜像。 -默认情况下，所有用于[配置 Kubernetes 集群](../../../../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)或启动 Rancher 中的工具（如监控，流水线，告警等）的镜像都是从 Docker Hub 中拉取的。在 Rancher 的离线安装中，你需要一个私有仓库，该仓库位于你的 Rancher Server 中某个可访问的位置。然后，你可加载该存有所有镜像的镜像仓库。 +默认情况下，所有用于[配置 Kubernetes 集群](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)或启动 Rancher 中的工具（如监控，流水线，告警等）的镜像都是从 Docker Hub 中拉取的。在 Rancher 的离线安装中，你需要一个私有仓库，该仓库位于你的 Rancher Server 中某个可访问的位置。然后，你可加载该存有所有镜像的镜像仓库。 使用 Docker 安装 Rancher，和把 Rancher 安装到 Kubernetes 集群，其对应的推送镜像到私有镜像仓库步骤是一样的。 -你使用 Rancher 配置的下游集群是否有运行 Windows 的节点，决定了本文涉及的步骤。我们提供的推送镜像到私有镜像仓库步骤，是基于假设 Rancher 仅配置运行 Linux 节点的下游 Kubernetes 集群的。但是，如果你计划[在下游 Kubernetes 集群中使用 Windows 节点](../../../../pages-for-subheaders/use-windows-clusters.md)，我们有单独的文档来介绍如何为需要的镜像提供支持。 +你使用 Rancher 配置的下游集群是否有运行 Windows 的节点，决定了本文涉及的步骤。我们提供的推送镜像到私有镜像仓库步骤，是基于假设 Rancher 仅配置运行 Linux 节点的下游 Kubernetes 集群的。但是，如果你计划[在下游 Kubernetes 集群中使用 Windows 节点](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)，我们有单独的文档来介绍如何为需要的镜像提供支持。 :::note 先决条件： diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/other-installation-methods.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/other-installation-methods.md new file mode 100644 index 00000000000..24d8ac7d47c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/other-installation-methods.md @@ -0,0 +1,19 @@ +--- +title: 其他安装方式 +--- + +### 离线安装 + +按照[以下步骤](air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)在离线环境中安装 Rancher Server。 + +离线环境可以是 Rancher Server 离线安装、防火墙后面或代理后面。 + +### Docker 安装 + +[单节点 Docker 安装](rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md)适用于想要测试 Rancher 的用户。你无需使用 Helm 在 Kubernetes 集群上运行 Rancher，你可以使用 `docker run` 命令，把 Rancher Server 组件安装到单个节点上。 + +Docker 安装仅用于开发和测试环境。 + +由于只有一个节点和一个 Docker 容器，因此，如果该节点发生故障，由于其他节点上没有可用的 etcd 数据副本，你将丢失 Rancher Server 的所有数据。 + +Rancher backup operator 可将 Rancher 从单个 Docker 容器迁移到高可用 Kubernetes 集群上。详情请参见[把 Rancher 迁移到新集群](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher.md index 63da5f840ae..7ac7caca50c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher.md @@ -6,6 +6,8 @@ title: 3. 安装 Rancher ### 安装 Helm CLI + + 在具有 kubeconfig 的主机上安装 [Helm](https://helm.sh/docs/intro/install/) CLI 以访问 Kubernetes 集群： ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md new file mode 100644 index 00000000000..1701dba329e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md @@ -0,0 +1,13 @@ +--- +title: 使用 HTTP 代理安装 Rancher +--- + +很多企业本地运行的服务器或虚拟机不能直接访问互联网，但是出于安全考虑，他们必须通过 HTTP(S) 代理连接到外部服务。本教程将分步介绍如何在这样的环境中进行高可用的 Rancher 安装。 + +另外，用户也可以在没有任何互联网访问的情况下离线设置 Rancher。详情请参见 [Rancher 官方文档](../air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)。 + +## 安装概要 + +1. [配置基础设施](set-up-infrastructure.md) +2. [配置 Kubernetes 集群](install-kubernetes.md) +3. [安装 Rancher](install-rancher.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure.md index 58414b8ed1d..f60e31b2c3e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure.md @@ -22,7 +22,7 @@ title: '1. 配置基础设施' 这些主机将通过 HTTP 代理连接到互联网。 -请确保你的节点满足[操作系统，容器运行时，硬件和网络](../../../../pages-for-subheaders/installation-requirements.md)的常规要求。 +请确保你的节点满足[操作系统，容器运行时，硬件和网络](../../installation-requirements/installation-requirements.md)的常规要求。 如需获取配置 Linux 节点的示例，请参见[在 Amazon EC2 中配置节点](../../../../how-to-guides/new-user-guides/infrastructure-setup/nodes-in-amazon-ec2.md)的教程。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md index 6e1ac8ee365..34ee707431b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md @@ -2,6 +2,8 @@ title: 证书故障排除 --- + + ### 如何确定我的证书格式是否为 PEM？ 你可以通过以下特征识别 PEM 格式： diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md new file mode 100644 index 00000000000..c2a63b86a5e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md @@ -0,0 +1,209 @@ +--- +title: 使用 Docker 将 Rancher 安装到单个节点中 +description: 在开发和测试环境中，你可以使用 Docker 安装。在单个 Linux 主机上安装 Docker，然后使用一个 Docker 容器部署 Rancher。 +--- + + + +Rancher 可以通过运行单个 Docker 容器进行安装。 + +在这种安装方案中，你需要将 Docker 安装到单个 Linux 主机，然后使用单个 Docker 容器将 Rancher 部署到主机中。 + +:::note 想要使用外部负载均衡器？ + +请参阅[使用外部负载均衡器的 Docker 安装](../../../../how-to-guides/advanced-user-guides/configure-layer-7-nginx-load-balancer.md)。 + +::: + +Rancher 的 Docker 安装仅推荐用于开发和测试环境中。Rancher 版本决定了能否将 Rancher 迁移到高可用集群。 + +Rancher backup operator 可将 Rancher 从单个 Docker 容器迁移到高可用 Kubernetes 集群上。详情请参见[把 Rancher 迁移到新集群](../../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)。 + +## Rancher 特权访问 + +当 Rancher Server 部署在 Docker 容器中时，容器内会安装一个本地 Kubernetes 集群供 Rancher 使用。为 Rancher 的很多功能都是以 deployment 的方式运行的，而在容器内运行容器是需要特权模式的，因此你需要在安装 Rancher 时添加 `--privileged` 选项。 + +## 操作系统，Docker，硬件和网络要求 + +请确保你的节点满足常规的[安装要求](../../installation-requirements/installation-requirements.md)。 + +## 1. 配置 Linux 主机 + +按照[要求](../../installation-requirements/installation-requirements.md)配置一个 Linux 主机，用于运行 Rancher Server。 + +## 2. 选择一个 SSL 选项并安装 Rancher + +出于安全考虑，使用 Rancher 时请使用 SSL（Secure Sockets Layer）。SSL 保护所有 Rancher 网络通信（如登录和与集群交互）的安全。 + +:::tip 你是否需要： + +- 使用代理。参见 [HTTP 代理配置](../../../../reference-guides/single-node-rancher-in-docker/http-proxy-configuration.md)。 +- 配置自定义 CA 根证书以访问服务。参见[自定义 CA 根证书](../../../../reference-guides/single-node-rancher-in-docker/advanced-options.md#自定义-ca-证书)。 +- 完成离线安装。参见 [离线：Docker 安装](../air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)。 +- 记录所有 Rancher API 的事务。参加 [API 审计](../../../../reference-guides/single-node-rancher-in-docker/advanced-options.md#api-审计日志)。 + +::: + +选择以下的选项之一： + +- [选项 A：使用 Rancher 生成的默认自签名证书](#选项-a使用-rancher-生成的默认自签名证书) +- [选项 B：使用你自己的证书 - 自签名](#选项-b使用你自己的证书---自签名) +- [选项 C：使用你自己的证书 - 可信 CA 签名的证书](#选项-c使用你自己的证书---可信-ca-签名的证书) +- [选项 D：Let's Encrypt 证书](#选项-dlets-encrypt-证书) +- [选项 E：Localhost 隧道，不使用证书](#选项-elocalhost-隧道不使用证书) + +### 选项 A：使用 Rancher 生成的默认自签名证书 + +如果你在不考虑身份验证的开发或测试环境中安装 Rancher，可以使用 Rancher 生成的自签名证书安装 Rancher。这种安装方式避免了自己生成证书的麻烦。 + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + --privileged \ + rancher/rancher:latest +``` + +### 选项 B：使用你自己的证书 - 自签名 +在你团队访问 Rancher Server 的开发或测试环境中，创建一个用于你的安装的自签名证书，以便团队验证他们对实例的连接。 + +:::note 先决条件： + +使用 [OpenSSL](https://www.openssl.org/) 或其他方法创建自签名证书。 + +- 证书文件的格式必须是 PEM。 +- 在你的证书文件中，包括链中的所有中间证书。你需要对你的证书进行排序，把你的证书放在最前面，后面跟着中间证书。如需查看示例，请参见[证书故障排除](../getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md)。 + +::: + +创建证书后，运行以下 Docker 命令以安装 Rancher。使用 `-v` 标志并提供证书的路径，以将证书挂载到容器中。 + +| 占位符 | 描述 | +| ------------------- | --------------------- | +| `` | 包含证书文件的目录的路径。 | +| `` | 完整证书链的路径。 | +| `` | 证书私钥的路径。 | +| `` | CA 证书的路径。 | + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + -v //:/etc/rancher/ssl/cert.pem \ + -v //:/etc/rancher/ssl/key.pem \ + -v //:/etc/rancher/ssl/cacerts.pem \ + --privileged \ + rancher/rancher:latest +``` + +### 选项 C：使用你自己的证书 - 可信 CA 签名的证书 + +在公开暴露应用的生产环境中，请使用由可信 CA 签名的证书，以避免用户收到证书安全警告。 + +不建议将 Docker 安装用于生产环境。这些说明仅适用于测试和开发。 + +:::note 先决条件： + +- 证书文件的格式必须是 PEM。 +- 在你的证书文件中，包括可信 CA 提供的所有中间证书。你需要对你的证书进行排序，把你的证书放在最前面，后面跟着中间证书。如需查看示例，请参见[证书故障排除](../getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md)。 + +::: + +获取证书后，运行以下 Docker 命令。 + +- 使用 `-v` 标志并提供证书的路径，以将证书挂载到容器中。因为你的证书是由可信的 CA 签名的，因此你不需要安装额外的 CA 证书文件。 +- 使用 `--no-cacerts` 作为容器的参数，以禁用 Rancher 生成的默认 CA 证书。 + +| 占位符 | 描述 | +| ------------------- | ----------------------------- | +| `` | 包含证书文件的目录的路径。 | +| `` | 完整证书链的路径。 | +| `` | 证书私钥的路径。 | + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + -v //:/etc/rancher/ssl/cert.pem \ + -v //:/etc/rancher/ssl/key.pem \ + --privileged \ + rancher/rancher:latest \ + --no-cacerts +``` + +### 选项 D：Let's Encrypt 证书 + +:::caution + +Let's Encrypt 对新证书请求有频率限制。因此，请限制创建或销毁容器的频率。详情请参见 [Let's Encrypt 官方文档 - 频率限制](https://letsencrypt.org/docs/rate-limits/)。 + +::: + +你也可以在生产环境中使用 [Let's Encrypt](https://letsencrypt.org/) 证书。Let's Encrypt 使用 HTTP-01 质询来验证你对域名的控制权。如果要确认你对该域名有控制权，你可将用于访问 Rancher 的主机名（例如 `rancher.mydomain.com`）指向运行的主机的 IP。你可通过在 DNS 中创建 A 记录，以将主机名绑定到 IP 地址。 + +不建议将 Docker 安装用于生产环境。这些说明仅适用于测试和开发。 + +:::note 先决条件： + +- Let's Encrypt 是联网服务。因此，在内网和离线环境中不能使用。 +- 在 DNS 中创建一条记录，将 Linux 主机 IP 地址绑定到要用于访问 Rancher 的主机名（例如，`rancher.mydomain.com`）。 +- 在 Linux 主机上打开 `TCP/80` 端口。Let's Encrypt 的 HTTP-01 质询可以来自任何源 IP 地址，因此端口 `TCP/80` 必须开放开所有 IP 地址。 + +::: + +满足先决条件后，你可以运行以下命令使用 Let's Encrypt 证书安装 Rancher。 + +| 占位符 | 描述 | +| ----------------- | ------------------- | +| `` | 你的域名地址 | + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + --privileged \ + rancher/rancher:latest \ + --acme-domain +``` + +### 选项 E：Localhost 隧道，不使用证书 + +如果你在开发或测试环境中安装 Rancher，且环境中有运行的 localhost 隧道解决方案（如 [ngrok](https://ngrok.com/)），不要生成证书。此安装选项不需要证书。 + +- 使用 `--no-cacerts` 作为参数，以禁用 Rancher 生成的默认 CA 证书。 + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + --privileged \ + rancher/rancher:latest \ + --no-cacerts +``` + +## 高级选项 + +使用 Docker 将 Rancher 安装到单个节点时，有如下几个可开启的高级选项： + +- 自定义 CA 证书 +- API 审计日志 +- TLS 设置 +- 离线环境 +- 持久化数据 +- 在同一个节点中运行 `rancher/rancher` 和 `rancher/rancher-agent` + +详情请参见[本页](../../../../reference-guides/single-node-rancher-in-docker/advanced-options.md)。 + +## 故障排除 + +如需了解常见问题及故障排除提示，请参见[本页](certificate-troubleshooting.md)。 + +## 后续操作 + +- **推荐**：检查单节点[备份](../../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-docker-installed-rancher.md)和[恢复](../../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-docker-installed-rancher.md)。你可能暂时没有需要备份的数据，但是我们建议你在常规使用 Rancher 后创建备份。 +- 创建 Kubernetes 集群：[配置 Kubernetes 集群](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/roll-back-docker-installed-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/roll-back-docker-installed-rancher.md index 32fbf2f4254..762ec2bd1a5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/roll-back-docker-installed-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/roll-back-docker-installed-rancher.md @@ -2,6 +2,8 @@ title: 回滚 Docker 安装的 Rancher --- + + 如果 Rancher 升级没有成功完成，你需要回滚到你在 [Docker 升级](upgrade-docker-installed-rancher.md)之前使用的 Rancher 设置。回滚可以恢复： - 先前版本的 Rancher。 @@ -78,7 +80,7 @@ docker pull rancher/rancher: --privileged \ rancher/rancher: ``` - 特权访问是[必须](../../../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#rancher-特权访问)的。 + 特权访问是[必须](rancher-on-a-single-node-with-docker.md#rancher-特权访问)的。 :::danger diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md index 68d23b34c33..7dbef6d2cab 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md @@ -13,7 +13,7 @@ title: 升级 Docker 安装的 Rancher ## 先决条件 - 在 Rancher 文档中**检查[已知升级问题](../../install-upgrade-on-a-kubernetes-cluster/upgrades.md#已知升级问题)**，了解升级 Rancher 时最需要注意的问题。你可以在 [GitHub](https://github.com/rancher/rancher/releases) 发布说明以及 [Rancher 论坛](https://forums.rancher.com/c/announcements/12)中找到每个 Rancher 版本的已知问题。不支持升级或升级到 [rancher-alpha 仓库](../../resources/choose-a-rancher-version.md#helm-chart-仓库)中的任何 Chart。 -- **[仅适用于离线安装](../../../../pages-for-subheaders/air-gapped-helm-cli-install.md)：为新的 Rancher Server 版本收集和推送镜像**。按照指南为你想要升级的目标 Rancher 版本[推送镜像到私有镜像仓库](../air-gapped-helm-cli-install/publish-images.md)。 +- **[仅适用于离线安装](../air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)：为新的 Rancher Server 版本收集和推送镜像**。按照指南为你想要升级的目标 Rancher 版本[推送镜像到私有镜像仓库](../air-gapped-helm-cli-install/publish-images.md)。 ## 占位符 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/add-tls-secrets.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/add-tls-secrets.md index dc58f216be0..87f320d0dca 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/add-tls-secrets.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/add-tls-secrets.md @@ -42,4 +42,4 @@ Rancher 启动时会检索配置的 `tls-ca` 密文。如果 Rancher 在运行 ## 更新私有 CA 证书 -按照[步骤](update-rancher-certificate.md)更新 [Rancher 高可用 Kubernetes 安装](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md)中的 Ingress，或从默认自签名证书切换到自定义证书。 \ No newline at end of file +按照[步骤](update-rancher-certificate.md)更新 [Rancher 高可用 Kubernetes 安装](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中的 Ingress，或从默认自签名证书切换到自定义证书。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md index b168ecdcb05..7ddfcef2d8e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md @@ -11,7 +11,7 @@ title: 选择 Rancher 版本 -如果 Rancher Server 是[安装在 Kubernetes 集群上](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md)的，Rancher Server 的安装，升级和回滚中，都是使用 Kubernetes 集群上的 Helm Chart 来安装 Rancher 的。因此，在准备安装或升级 Rancher 高可用时，必须添加包含用于安装 Rancher 的 Chart 的 Helm Chart 仓库。 +如果 Rancher Server 是[安装在 Kubernetes 集群上](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)的，Rancher Server 的安装，升级和回滚中，都是使用 Kubernetes 集群上的 Helm Chart 来安装 Rancher 的。因此，在准备安装或升级 Rancher 高可用时，必须添加包含用于安装 Rancher 的 Chart 的 Helm Chart 仓库。 请参见 [Helm 版本要求](helm-version-requirements.md)选择 Helm 版本来安装 Rancher。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/helm-version-requirements.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/helm-version-requirements.md index 273711a4de0..9a1fc4d1078 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/helm-version-requirements.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/helm-version-requirements.md @@ -4,7 +4,9 @@ title: Helm 版本要求 本文介绍 Helm 的要求。Helm 是用于把 Rancher 安装在高可用 Kubernetes 集群上的工具。 -> 我们已针对 Helm 3 更新了安装指南。如果你使用 Helm 2 进行安装，请参见 [Helm 2 迁移到 Helm 3 文档](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/)。[本文](/versioned_docs/version-2.0-2.4/pages-for-subheaders/helm2.md)提供了较早的使用 Helm 2 的 Rancher 高可用安装指南的副本。如果你如果无法升级到 Helm 3，可以使用这个说明安装。 +> 我们已针对 Helm 3 更新了安装指南。如果你使用 Helm 2 进行安装，请参见 [Helm 2 迁移到 Helm 3 文档](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/)。[本文](/versioned_docs/version-2.0-2.4/getting-started/installation-and-upgrade/advanced-options/advanced-use-cases/helm2/helm2.md)提供了较早的使用 Helm 2 的 Rancher 高可用安装指南的副本。如果你如果无法升级到 Helm 3，可以使用这个说明安装。 + + - 如需安装或升级 Rancher 2.5，请使用 Helm 3.2.x 或更高版本。 - Kubernetes 1.16 要求 Helm 2.16.0 或更高版本。如果使用的是默认 Kubernetes 版本，请参见[发行说明](https://github.com/rancher/rke/releases)获取所使用的 RKE 版本。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/local-system-charts.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/local-system-charts.md index 4048c5b2566..17c965ad095 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/local-system-charts.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/local-system-charts.md @@ -10,4 +10,4 @@ title: 离线安装中设置本地 System Charts `system-charts` 的一个本地副本已经打包到 `rancher/rancher` 容器中。为了在离线安装中使用这些功能，你需要使用额外的环境变量 `CATTLE_SYSTEM_CATALOG=bundled` 来运行 Rancher 安装命令，该环境变量告诉 Rancher 使用 Chart 的本地副本，而不是尝试从 GitHub 获取 Chart。 -带有 `system-charts` 的 Rancher 安装命令示例包含在 Docker 和 Helm 的[离线安装说明](../../../pages-for-subheaders/air-gapped-helm-cli-install.md)中。 +带有 `system-charts` 的 Rancher 安装命令示例包含在 Docker 和 Helm 的[离线安装说明](../other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)中。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/resources.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/resources.md new file mode 100644 index 00000000000..2862aba2cd2 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/resources.md @@ -0,0 +1,25 @@ +--- +title: 资源 +--- + +### Docker 安装 + +[单节点 Docker 安装](../other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md)适用于想要测试 Rancher 的用户。你无需使用 Helm 在 Kubernetes 集群上运行 Rancher，你可以使用 `docker run` 命令，把 Rancher Server 组件安装到单个节点上。 + +由于只有一个节点和一个 Docker 容器，因此，如果该节点发生故障，由于其他节点上没有可用的 etcd 数据副本，你将丢失 Rancher Server 的所有数据。 + +### 离线安装 + +按照[以下步骤](../other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)在离线环境中安装 Rancher Server。 + +离线环境可以是 Rancher Server 离线安装、防火墙后面或代理后面。 + +### 高级选项 + +安装 Rancher 时，有如下几个可开启的高级选项：每个安装指南中都提供了对应的选项。了解选项详情： + +- [自定义 CA 证书](custom-ca-root-certificates.md) +- [API 审计日志](../../../how-to-guides/advanced-user-guides/enable-api-audit-log.md) +- [TLS 设置](../installation-references/tls-settings.md) +- [etcd 配置](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md) +- [离线安装 Local System Chart](local-system-charts.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md index 3d11417cda6..4678487523e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md @@ -4,7 +4,7 @@ title: 更新 Rancher 证书 ## 更新私有 CA 证书 -按照以下步骤轮换[安装在 Kubernetes 集群上](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md)、由 Rancher 使用的 SSL 证书和私有 CA，或转用由私有 CA 签发的 SSL 证书。 +按照以下步骤轮换[安装在 Kubernetes 集群上](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)、由 Rancher 使用的 SSL 证书和私有 CA，或转用由私有 CA 签发的 SSL 证书。 步骤概述： diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/upgrade-cert-manager.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/upgrade-cert-manager.md index 1b0f76a1809..f8a6cb3d1a0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/upgrade-cert-manager.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/resources/upgrade-cert-manager.md @@ -2,6 +2,8 @@ title: 升级 Cert-Manager --- +Rancher 适配 API 版本 `cert-manager.io/v1` 并且在 cert-manager v1.13.1 版本上进行了测试。 + Rancher 使用 cert-manager 为 Rancher 高可用部署自动生成和续期 TLS 证书。从 2019 秋季开始，cert-manager 发生了以下的三个重要变更。如果你在此时间段前创建了 Rancher 高可用部署，请进行相关操作。 1. [从 2019 年 11 月 1 日开始，Let's Encrypt 已阻止低于 0.8.0 的 cert-manager 实例。](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753) @@ -139,6 +141,8 @@ Rancher 使用 cert-manager 为 Rancher 高可用部署自动生成和续期 TLS --set cainjector.image.repository=/quay.io/jetstack/cert-manager-cainjector ``` + + Helm 2 命令如下： ```plain @@ -260,7 +264,7 @@ cert-manager-webhook-787858fcdb-nlzsq 1/1 Running 0 2m --- -Rancher 现在支持 cert-manager 1.6.2 和 1.7.1。推荐使用 v1.7.x，因为 v 1.6.x 将在 2022 年 3 月 30 日结束生命周期。详情请参见 [cert-manager 文档](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#4-安装-cert-manager)。有关将 cert-manager 从 1.5 升级到 1.6 的说明，请参见上游的 [cert-manager 文档](https://cert-manager.io/docs/installation/upgrading/upgrading-1.5-1.6/)。有关将 cert-manager 从 1.6 升级到 1.7 的说明，请参见上游的 [cert-manager 文档](https://cert-manager.io/docs/installation/upgrading/upgrading-1.6-1.7/)。 +Rancher 现在支持 cert-manager 1.6.2 和 1.7.1。推荐使用 v1.7.x，因为 v 1.6.x 将在 2022 年 3 月 30 日结束生命周期。详情请参见 [cert-manager 文档](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md#4-安装-cert-manager)。有关将 cert-manager 从 1.5 升级到 1.6 的说明，请参见上游的 [cert-manager 文档](https://cert-manager.io/docs/installation/upgrading/upgrading-1.5-1.6/)。有关将 cert-manager 从 1.6 升级到 1.7 的说明，请参见上游的 [cert-manager 文档](https://cert-manager.io/docs/installation/upgrading/upgrading-1.6-1.7/)。 --- diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md index aa851f3a50c..b23f7651dd3 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md @@ -32,9 +32,9 @@ RKE v1.1.0 改变了集群升级的方式。 :::note 先决条件： -- 以下选项适用于 [Rancher 启动的 Kubernetes 集群](../../pages-for-subheaders/launch-kubernetes-with-rancher.md)和[注册的 K3s Kubernetes 集群](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/register-existing-clusters.md#已注册-rke2-和-k3s-集群的附加功能)。 +- 以下选项适用于 [Rancher 启动的 Kubernetes 集群](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)和[注册的 K3s Kubernetes 集群](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/register-existing-clusters.md#已注册-rke2-和-k3s-集群的附加功能)。 - 以下选项也适用于导入且已注册的 RKE2 集群。如果你从外部云平台导入集群但不注册，你将无法在 Rancher 中升级 Kubernetes 版本。 -- 在升级 Kubernetes 之前，先[备份你的集群](../../pages-for-subheaders/backup-restore-and-disaster-recovery.md)。 +- 在升级 Kubernetes 之前，先[备份你的集群](../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md)。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md index 2d9c0f00918..3d01c38aed8 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md @@ -10,7 +10,7 @@ Kubernetes API 可以在次要版本之间更改。因此，我们不支持引 ::: -Rancher 的 Kubernetes 元数据包含 Rancher 用于配置 [RKE 集群](../../pages-for-subheaders/launch-kubernetes-with-rancher.md)的 Kubernetes 版本信息。Rancher 会定期同步数据并为 **系统镜像**、**服务选项**和**插件模板**创建自定义资源定义 (CRD)。因此，当新的 Kubernetes 版本与 Rancher Server 版本兼容时，Kubernetes 元数据可以使 Rancher 使用新版本来配置集群。元数据概述了 [Rancher Kubernetes Engine](https://rancher.com/docs/rke/latest/en/) (RKE) 用于部署各种 Kubernetes 版本的信息。 +Rancher 的 Kubernetes 元数据包含 Rancher 用于配置 [RKE 集群](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)的 Kubernetes 版本信息。Rancher 会定期同步数据并为 **系统镜像**、**服务选项**和**插件模板**创建自定义资源定义 (CRD)。因此，当新的 Kubernetes 版本与 Rancher Server 版本兼容时，Kubernetes 元数据可以使 Rancher 使用新版本来配置集群。元数据概述了 [Rancher Kubernetes Engine](https://rancher.com/docs/rke/latest/en/) (RKE) 用于部署各种 Kubernetes 版本的信息。 下表描述了受周期性数据同步影响的 CRD。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/overview.md index 903c8c28bd2..be0cf58660c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/overview.md @@ -30,21 +30,21 @@ Rancher API Server 是基于嵌入式 Kubernetes API Server 和 etcd 数据库 ### 授权和基于角色的权限控制（RBAC） -- **用户管理**：Rancher API Server 除了管理本地用户，还[管理用户用来访问外部服务所需的认证信息](../pages-for-subheaders/authentication-config.md)，如登录 Active Directory 和 GitHub 所需的账号密码。 -- **授权**：Rancher API Server 可以管理[访问控制策略](../pages-for-subheaders/manage-role-based-access-control-rbac.md)和[安全策略](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md)。 +- **用户管理**：Rancher API Server 除了管理本地用户，还[管理用户用来访问外部服务所需的认证信息](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md)，如登录 Active Directory 和 GitHub 所需的账号密码。 +- **授权**：Rancher API Server 可以管理[访问控制策略](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md)和[安全策略](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md)。 ### 使用 Kubernetes 的功能 -- **配置 Kubernetes 集群**：Rancher API Server 可以在已有节点上[配置 Kubernetes](../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)，或进行 [Kubernetes 版本升级](installation-and-upgrade/upgrade-and-roll-back-kubernetes.md)。 -- **管理应用商店**：Rancher 支持使用 [Helm Chart 应用商店](../pages-for-subheaders/helm-charts-in-rancher.md)实现轻松重复部署应用。 -- **管理项目**：项目由集群中多个命名空间和访问控制策略组成，是 Rancher 中的一个概念，Kubernetes 中并没有这个概念。你可以使用项目实现以组为单位，管理多个命名空间，并进行 Kubernetes 相关操作。Rancher UI 提供用于[项目管理](../pages-for-subheaders/manage-projects.md)和[项目内应用管理](../pages-for-subheaders/kubernetes-resources-setup.md)的功能。 -- **Fleet 持续交付**：在 Rancher 中，你可以使用 [Fleet 持续交付](../pages-for-subheaders/fleet-gitops-at-scale.md)将应用程序从 Git 仓库部署到目标下游 Kubernetes 集群，无需任何手动操作。 -- **Istio**：[Rancher 与 Istio 集成](../pages-for-subheaders/istio.md)，使得管理员或集群所有者可以将 Istio 交给开发者，然后开发者使用 Istio 执行安全策略，排查问题，或为蓝绿部署，金丝雀部署，和 A/B 测试进行流量管理。 +- **配置 Kubernetes 集群**：Rancher API Server 可以在已有节点上[配置 Kubernetes](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)，或进行 [Kubernetes 版本升级](installation-and-upgrade/upgrade-and-roll-back-kubernetes.md)。 +- **管理应用商店**：Rancher 支持使用 [Helm Chart 应用商店](../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md)实现轻松重复部署应用。 +- **管理项目**：项目由集群中多个命名空间和访问控制策略组成，是 Rancher 中的一个概念，Kubernetes 中并没有这个概念。你可以使用项目实现以组为单位，管理多个命名空间，并进行 Kubernetes 相关操作。Rancher UI 提供用于[项目管理](../how-to-guides/advanced-user-guides/manage-projects/manage-projects.md)和[项目内应用管理](../how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md)的功能。 +- **Fleet 持续交付**：在 Rancher 中，你可以使用 [Fleet 持续交付](../integrations-in-rancher/fleet/fleet.md)将应用程序从 Git 仓库部署到目标下游 Kubernetes 集群，无需任何手动操作。 +- **Istio**：[Rancher 与 Istio 集成](../integrations-in-rancher/istio/istio.md)，使得管理员或集群所有者可以将 Istio 交给开发者，然后开发者使用 Istio 执行安全策略，排查问题，或为蓝绿部署，金丝雀部署，和 A/B 测试进行流量管理。 ### 配置云基础设施 - **同步节点信息**：Rancher API Server 可以同步所有集群中全部[节点](../how-to-guides/new-user-guides/manage-clusters/nodes-and-node-pools.md)的信息。 -- **配置云基础设施**：如果你为 Rancher 配置了云提供商，Rancher 可以在云端动态配置[新节点](../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md)和[持久化存储](../pages-for-subheaders/create-kubernetes-persistent-storage.md)。 +- **配置云基础设施**：如果你为 Rancher 配置了云提供商，Rancher 可以在云端动态配置[新节点](../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)和[持久化存储](../how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md)。 ### 查看集群信息 @@ -54,9 +54,9 @@ Rancher API Server 是基于嵌入式 Kubernetes API Server 和 etcd 数据库 ## 使用 Rancher 编辑下游集群 -对于已有集群而言，可提供的选项和设置取决于你配置集群的方法。例如，只有[通过 RKE 启动](../pages-for-subheaders/launch-kubernetes-with-rancher.md)的集群才有可编辑的**集群选项**。 +对于已有集群而言，可提供的选项和设置取决于你配置集群的方法。例如，只有[通过 RKE 启动](../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)的集群才有可编辑的**集群选项**。 -使用 Rancher 创建集群后，集群管理员可以管理集群成员，管理节点池，或进行[其他操作](../pages-for-subheaders/cluster-configuration.md)。 +使用 Rancher 创建集群后，集群管理员可以管理集群成员，管理节点池，或进行[其他操作](../reference-guides/cluster-configuration/cluster-configuration.md)。 下表总结了每一种类型的集群和对应的可编辑的选项和设置： diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/aws-marketplace.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/aws-marketplace.md index 2fc1ce0bd33..1d06135d7f4 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/aws-marketplace.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/aws-marketplace.md @@ -1,6 +1,6 @@ --- -title: Rancher AWS Marketplace 快速入门 -description: 使用 Amazon EKS 部署 Rancher Server。 +title: Rancher Prime AWS Marketplace 快速入门 +description: 使用 Amazon Marketplace 列表部署 Rancher Server。 --- -Amazon Elastic Kubernetes Service (EKS) 可以快速[将 Rancher 部署到 Amazon Web Services (AWS)](https://documentation.suse.com/trd/kubernetes/single-html/gs_rancher_aws-marketplace/)。详情请参见我们的 [Amazon Marketplace 列表](https://aws.amazon.com/marketplace/pp/prodview-go7ent7goo5ae)。观看 [demo](https://youtu.be/9dznJ7Ons0M)，了解 AWS Marketplace SUSE Rancher 设置的演练。 +你可以快速将 Rancher Prime 部署到 Amazon Elastic Kubernetes Service (EKS)。详情请参见[说明](https://suse-enceladus.github.io/marketplace-docs/rancher-prime/aws/?repository=rancher-payg-billing-adapter-llc-prd)以及我们的 [Amazon Marketplace 列表](https://aws.amazon.com/marketplace/pp/prodview-go7ent7goo5ae)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/aws.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/aws.md index cb47333ce32..34590ac8931 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/aws.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/aws.md @@ -7,7 +7,7 @@ description: 阅读此分步 Rancher AWS 指南，以快速部署带有单节点 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -86,7 +86,7 @@ AWS 模块只创建一个 EC2 密钥对、一个 EC2 安全组和一个 EC2 实 ## 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/azure.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/azure.md index 33a8d411ba0..2775340b7a5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/azure.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/azure.md @@ -7,7 +7,7 @@ description: 阅读此分步 Rancher Azure 指南，以快速部署带有单节 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -72,7 +72,7 @@ description: 阅读此分步 Rancher Azure 指南，以快速部署带有单节 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager.md new file mode 100644 index 00000000000..5eebb87e4e8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager.md @@ -0,0 +1,20 @@ +--- +title: 部署 Rancher Server +--- + +你可使用以下指南之一，在你选择的提供商中部署和配置 Rancher 和 Kubernetes 集群。 + +- [AWS](aws.md)（使用 Terraform） +- [AWS Marketplace](aws-marketplace.md)（使用 Amazon EKS） +- [Azure](azure.md)（使用 Terraform） +- [DigitalOcean](digitalocean.md)（使用 Terraform） +- [GCP](gcp.md)（使用 Terraform） +- [Hetzner Cloud](hetzner-cloud.md)（使用 Terraform） +- [Linode](linode.md) (使用 Terraform) +- [Vagrant](vagrant.md) +- [Equinix Metal](equinix-metal.md) +- [Outscale](outscale-qs.md)（使用 Terraform） + +如有需要，你可以查看以下指南以了解分步步骤。如果你需要在其他提供商中或本地运行 Rancher，或者你只是想看看它是多么容易上手，你可阅读以下指南： + +- [手动安装](helm-cli.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md index e87f17c26d0..8e0280f9fa6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md @@ -7,7 +7,7 @@ description: 阅读此分步 Rancher DigitalOcean 指南，以快速部署带有 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -65,7 +65,7 @@ description: 阅读此分步 Rancher DigitalOcean 指南，以快速部署带有 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/equinix-metal.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/equinix-metal.md index c7316352cc2..bca4519d296 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/equinix-metal.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/equinix-metal.md @@ -11,7 +11,7 @@ title: Rancher Equinix Metal 快速入门 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。不建议将 Docker 安装用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。不建议将 Docker 安装用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -38,7 +38,7 @@ title: Rancher Equinix Metal 快速入门 - 如果使用 CLI 或 API 配置新的 Equinix Metal Server，你需要提供项目 ID、计划、metro 和操作系统。 - 当使用云主机的虚拟机时，你需要允许 80 和 443 端口的入站 TCP 通信。有关端口配置的信息，请参见你的云主机的文档。 - 如需了解所有端口要求，请参见 [Docker 安装](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)。 -- 根据我们的[要求](../../../pages-for-subheaders/installation-requirements.md)配置主机。 +- 根据我们的[要求](../../installation-and-upgrade/installation-requirements/installation-requirements.md)配置主机。 ::: ### 2. 安装 Rancher @@ -103,4 +103,4 @@ title: Rancher Equinix Metal 快速入门 #### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md index 5c113fc447c..74af3068f04 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md @@ -7,7 +7,7 @@ description: 阅读此分步 Rancher GCP 指南，以快速部署带有单节点 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -68,7 +68,7 @@ description: 阅读此分步 Rancher GCP 指南，以快速部署带有单节点 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md index f3ecac4d210..b4f0395ac2d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md @@ -10,7 +10,7 @@ title: Helm CLI 快速入门 你的 Linux 主机可以位于任何地方。例如，它可以是 Amazon EC2 实例、Digital Ocean Droplet 或 Azure 虚拟机。其他 Rancher 文档也经常称它们为“节点”。部署 Linux 主机的一种方法是设置一个 Amazon EC2 实例，如[本教程](../../../how-to-guides/new-user-guides/infrastructure-setup/nodes-in-amazon-ec2.md)中所示。 -完整的安装要求在[这里](../../../pages-for-subheaders/installation-requirements.md)。 +完整的安装要求在[这里](../../installation-and-upgrade/installation-requirements/installation-requirements.md)。 ## 在 Linux 上安装 K3s @@ -147,6 +147,6 @@ helm install rancher rancher-latest/rancher ` 为了简化说明，我们使用了一个假域名和自签名证书来进行安装。因此，你可能需要在 Web 浏览器中添加一个安全例外来查看 Rancher UI。请注意，对于生产安装，你需要具有负载均衡器、真实域名和真实证书的高可用性设置。 -这些说明还省略了完整的安装要求和其他安装选项。如果你对这些步骤有任何疑问，请参阅完整的 [Helm CLI 安装文档](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md)。 +这些说明还省略了完整的安装要求和其他安装选项。如果你对这些步骤有任何疑问，请参阅完整的 [Helm CLI 安装文档](../../installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)。 -要使用新的 Rancher Server 来启动新的 Kubernetes 集群，你可能需要在 Rancher 中设置云凭证。有关更多信息，请参阅[使用 Rancher 启动 Kubernetes 集群](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md)。 +要使用新的 Rancher Server 来启动新的 Kubernetes 集群，你可能需要在 Rancher 中设置云凭证。有关更多信息，请参阅[使用 Rancher 启动 Kubernetes 集群](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md index 585353441b2..16ebb833b2a 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md @@ -7,7 +7,7 @@ description: 阅读此分步 Rancher Hetzner Cloud 指南，以快速部署带 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -67,7 +67,7 @@ description: 阅读此分步 Rancher Hetzner Cloud 指南，以快速部署带 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/linode.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/linode.md new file mode 100644 index 00000000000..9b6a00f6f7e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/linode.md @@ -0,0 +1,79 @@ +--- +title: Rancher Linode 快速入门指南 +description: 阅读此分步 Rancher Linode 指南，以快速部署带有单节点下游 Kubernetes 集群的 Rancher Server。 +--- + +你可以参考以下步骤，在 Linode 的单节点 K3s Kubernetes 集群中快速部署 Rancher Server，并附加一个单节点下游 Kubernetes 集群。 + +:::caution + +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 + +::: + +## 先决条件 + +:::caution + +部署到 Linode 会产生费用。 + +::: + +- [Linode 账号](https://linode.com): 用于运行服务器和集群。 +- [Linode 访问密钥](https://www.linode.com/docs/products/tools/api/guides/manage-api-tokens/): 用于权限认证的 Linode 访问密钥。 +- [Terraform](https://www.terraform.io/downloads.html): 用于在 Linode 中配置服务器和集群。 + + +## 开始使用 + +1. 使用命令行工具，执行 `git clone https://github.com/rancher/quickstart` 把 [Rancher Quickstart](https://github.com/rancher/quickstart) 克隆到本地。 + +2. 执行 `cd quickstart/rancher/linode` 命令，进入包含 Terraform 文件的 Linode 文件夹。 + +3. 把 `terraform.tfvars.example` 文件重命名为 `terraform.tfvars`。 + +4. 编辑 `terraform.tfvars` 文件，并替换以下变量： + - `linode_token` - 上面提到的 Linode 访问密钥。 + - `rancher_server_admin_password` - 替换为创建 Rancher Server 的 admin 账号的密码（最少 12 字符） + +5. **可选**：修改 `terraform.tfvars` 中的可选参数。 + 参见 [Quickstart Readme](https://github.com/rancher/quickstart) 以及 [Linode Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/linode) 了解更多信息。 + 建议包括： + - `linode_region` - 创建服务器以及集群的目标 Linode 区域。 + - 默认: `eu-central` + - 完整的区域列表, 请参照[官方的可用区域页面](https://www.linode.com/global-infrastructure/availability/). + - `prefix` - 所有创建资源的前缀 + - `linode_type` - 所有的 Linode 资源使用的类型/计划 + - 默认: `g6-standard-2` + - 完整的计划列表, 请参照[官方的计划类型页面](https://www.linode.com/docs/products/compute/compute-instances/plans/). + +6. 执行 `terraform init`。 + +7. 执行 `terraform apply --auto-approve` 以初始化环境。然后，等待命令行工具返回以下信息： + + ``` + Apply complete! Resources: 15 added, 0 changed, 0 destroyed. + + Outputs: + + rancher_node_ip = xx.xx.xx.xx + rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io + workload_node_ip = yy.yy.yy.yy + ``` + +8. 将以上输出中的 `rancher_server_url` 粘贴到浏览器中。在登录页面中登录（默认用户名为 `admin`，密码为在 `rancher_server_admin_password` 中设置的密码）。 +9. 使用 `quickstart/rancher/linode` 中生成的 `id_rsa` 密钥 SSH 到 Rancher Server。 + +#### 结果 + +两个 Kubernetes 集群已部署到你的 Linode 账户中，一个运行 Rancher Server，另一个为实验部署做好准备。请注意，虽然这种设置是探索 Rancher 功能的好方法，但在生产环境中，应遵循我们的高可用设置指南。用于虚拟机的 SSH 密钥是自动生成的，存储在模块目录中。 + +### 后续操作 + +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 + +## 销毁环境 + +1. 进入 `quickstart/rancher/linode` 文件夹，然后执行 `terraform destroy --auto-approve`。 + +2. 等待命令行界面显示资源已删除的消息。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md index 46b42901db9..76d320a862e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md @@ -7,7 +7,7 @@ description: 阅读此分步 Rancher Outscale 指南，以快速部署带有单 :::note -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -67,7 +67,7 @@ description: 阅读此分步 Rancher Outscale 指南，以快速部署带有单 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/vagrant.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/vagrant.md index dcddb74cfe9..1622c44a7b0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/vagrant.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-rancher-manager/vagrant.md @@ -6,7 +6,7 @@ title: Vagrant 快速入门 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -42,7 +42,7 @@ title: Vagrant 快速入门 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md new file mode 100644 index 00000000000..80c8505d35e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md @@ -0,0 +1,8 @@ +--- +title: 部署工作负载 +--- + +这些指南指导你完成一个应用的部署，包括如何将应用暴露在集群之外使用。 + +- [部署带有 Ingress 的工作负载](workload-ingress.md) +- [部署带有 NodePort 的工作负载](nodeports.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/quick-start-guides.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/quick-start-guides.md new file mode 100644 index 00000000000..e3e750dc9b4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/quick-start-guides.md @@ -0,0 +1,17 @@ +--- +title: Rancher 部署快速入门指南 +--- + +:::caution + +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../installation-and-upgrade/installation-and-upgrade.md)。 + +::: + +你可以阅读本章节，以快速开始部署和测试 Rancher 2.x。本章节包含 Rancher 的简单设置和一些常见用例的说明。未来，我们会在本章节中添加更多内容。 + +我们提供以下快速入门指南： + +- [部署 Rancher Server](deploy-rancher-manager/deploy-rancher-manager.md)：使用最方便的方式运行 Rancher。 + +- [部署工作负载](deploy-workloads/deploy-workloads.md)：部署一个简单的[工作负载](https://kubernetes.io/docs/concepts/workloads/)并公暴露工作负载，以从集群外部访问工作负载。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades.md index dee4c863bcf..8e5c0eca63f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades.md @@ -1,5 +1,5 @@ --- -title: 在离线环境中渲染 Helm 模板 +title: 在离线环境中升级 --- :::note @@ -35,6 +35,26 @@ helm template rancher ./rancher-.tgz --output-dir . \ --set useBundledSystemChart=true # Use the packaged Rancher system charts ``` +#### 解决 UPGRADE FAILED 错误 + +如果你遇到错误消息 `Error: UPGRADE FAILED: "rancher" has no deployed releases`，Rancher 可能是通过 `helm template` 命令安装的。要成功升级 Rancher，请改用以下命令： + +``` +helm template rancher ./rancher-.tgz --output-dir . \ + --no-hooks \ # prevent files for Helm hooks from being generated + --namespace cattle-system \ + --set hostname= \ + --set certmanager.version= \ + --set rancherImage=/rancher/rancher \ + --set systemDefaultRegistry= \ # Set a default private registry to be used in Rancher + --set useBundledSystemChart=true # Use the packaged Rancher system charts +``` + +执行 Helm 命令后，需要应用渲染后的模板： + +``` +kubectl -n cattle-system apply -R -f ./rancher +``` ### 选项 B：使用 Kubernetes 密文从文件中获取证书 ```plain @@ -62,16 +82,6 @@ helm template rancher ./rancher-.tgz --output-dir . \ --set useBundledSystemChart=true # Use the packaged Rancher system charts ``` -### 应用已渲染的模板 - -将渲染的 manifest 目录复制到可以访问 Rancher Server 集群的系统中，并应用渲染的模板。 - -使用 `kubectl` 来应用渲染的 manifest。 - -```plain -kubectl -n cattle-system apply -R -f ./rancher -``` - ## 验证升级 登录 Rancher 以确认升级成功。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md new file mode 100644 index 00000000000..437bc35815e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md @@ -0,0 +1,351 @@ +--- +title: 在 Kubernetes 集群上安装/升级 Rancher +description: 了解如何在开发和生产环境中安装 Rancher。了解单节点和高可用安装 +--- + +在本节中，你将学习如何使用 Helm CLI 在 Kubernetes 集群上部署 Rancher。 + + +## 先决条件 + +- [Kubernetes 集群](#kubernetes-集群) +- [Ingress Controller](#ingress-controller) +- [CLI 工具](#cli-工具) + +### Kubernetes 集群 + +设置 Rancher Server 的本地 Kubernetes 集群。 + +Rancher 可以安装在任何 Kubernetes 集群上。这个集群可以使用上游 Kubernetes，也可以使用 Rancher 的 Kubernetes 发行版之一，也可以是来自 Amazon EKS 等提供商的托管 Kubernetes 集群。 + +你可参考以下教程，以获得设置 Kubernetes 集群的帮助： + +- **RKE**：[安装 RKE Kubernetes 集群的教程](../../../how-to-guides/new-user-guides/kubernetes-cluster-setup/rke1-for-rancher.md)；[为高可用 RKE 集群设置基础设施的教程](../../../how-to-guides/new-user-guides/infrastructure-setup/ha-rke1-kubernetes-cluster.md)。 +- **K3s**：[安装 K3s Kubernetes 集群的教程](../../../how-to-guides/new-user-guides/kubernetes-cluster-setup/k3s-for-rancher.md)；[设置高可用 K3s 集群的基础设施的教程](../../../how-to-guides/new-user-guides/infrastructure-setup/ha-k3s-kubernetes-cluster.md)。 +- **RKE2:** ：[安装 RKE2 Kubernetes 集群的教程](../../../how-to-guides/new-user-guides/kubernetes-cluster-setup/rke2-for-rancher.md)；[设置高可用 RKE2 集群的基础设施的教程](../../../how-to-guides/new-user-guides/infrastructure-setup/ha-rke2-kubernetes-cluster.md)。 +- **Amazon EKS**：[在 Amazon EKS 上安装 Rancher 以及如何安装 Ingress Controller 以访问 Rancher Server](rancher-on-amazon-eks.md)。 +- **AKS**：[使用 Azure Kubernetes 服务安装 Rancher 以及如何安装 Ingress Controller 以访问 Rancher Server](rancher-on-aks.md)。 +- **GKE**：有关如何使用 GKE 安装 Rancher，包括如何安装 Ingress Controller 以便可以访问 Rancher Server，请参阅[此页面](rancher-on-gke.md)。GKE 创建 Kubernetes 集群时有两种运行模式，分别是 Autopilot 和 Standard 模式。Autopilot 模式的集群配置对编辑 kube-system 命名空间有限制。但是，Rancher 在安装时需要在 kube-system 命名空间中创建资源。因此，你将无法在以 Autopilot 模式创建的 GKE 集群上安装 Rancher。 + + +### Ingress Controller + +Rancher UI 和 API 通过 Ingress 公开。换言之，安装 Rancher 的 Kubernetes 集群必须包含一个 Ingress Controller。 + +对于 RKE、RKE2 和 K3s，你不需要手动安装 Ingress Controller，因为它是默认安装的。 + +对于默认不包含 Ingress Controller 的发行版（例如 EKS、GKE 或 AKS 等托管 Kubernetes 集群），你必须先部署 Ingress Controller。请注意，Rancher Helm Chart 默认情况下不会在 Ingress 上设置 `ingressClassName`。因此，你必须将 Ingress Controller 配置为在没有 `ingressClassName` 的情况下也可以监视 Ingress。 + +上面的 **Amazon EKS**、**AKS** 和 **GKE** 教程中包含了示例。 + +### CLI 工具 + +设置 Kubernetes 集群需要以下 CLI 工具。请确保这些工具已安装并在你的 `$PATH` 中可用。 + +- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl) - Kubernetes 命令行工具。 +- [Helm](https://docs.helm.sh/using_helm/#installing-helm) - Kubernetes 的包管理器。请参见 [Helm 版本要求](../resources/helm-version-requirements.md)选择 Helm 版本来安装 Rancher。请为你的具体平台参见 [Helm 项目提供的说明](https://helm.sh/docs/intro/install/)。 + +## 安装 Rancher Helm Chart + +Rancher 是使用 Kubernetes 的 [Helm](https://helm.sh/) 包管理器安装的。Helm Chart 为 Kubernetes YAML 清单文件提供了模板语法。通过 Helm，用户可以创建可配置的 deployment，而不仅仅只能使用静态文件。 + +如果系统无法直接访问互联网，请参见[离线环境：Kubernetes 安装](../other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha.md)。 + +如果要指定安装的 Rancher 版本，请参见[选择 Rancher 版本](../resources/choose-a-rancher-version.md)。 + +如果要指定用于安装 Rancher 的 Helm 版本，请参见[Helm 版本要求](../resources/helm-version-requirements.md)。 + +:::note + +本安装指南假定你使用的是 Helm 3。 + +::: + +要设置 Rancher： + +1. [添加 Helm Chart 仓库](#1-添加-helm-chart-仓库) +2. [为 Rancher 创建命名空间](#2-为-rancher-创建命名空间) +3. [选择 SSL 配置](#3-选择-ssl-配置) +4. [安装 cert-manager](#4-安装-cert-manager)（除非你自带证书，否则 TLS 将在负载均衡器上终止） +5. [使用 Helm 和你选择的证书选项安装 Rancher](#5-根据你选择的证书选项通过-helm-安装-rancher) +6. [验证 Rancher Server 是否部署成功](#6-验证-rancher-server-是否部署成功) +7. [保存选项](#7-保存选项) + +### 1. 添加 Helm Chart 仓库 + +执行 `helm repo add` 命令，以添加包含安装 Rancher 的 Chart 的 Helm Chart 仓库。有关如何选择仓库，以及哪个仓库最适合你的用例，请参见[选择 Rancher 版本](../getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md)。 + +- Latest：建议用于试用最新功能 + ``` + helm repo add rancher-latest https://releases.rancher.com/server-charts/latest + ``` +- Stable：建议用于生产环境 + ``` + helm repo add rancher-stable https://releases.rancher.com/server-charts/stable + ``` +- Alpha：即将发布的实验性预览。 + ``` + helm repo add rancher-alpha https://releases.rancher.com/server-charts/alpha + ``` + 注意：不支持升级到 Alpha 版、从 Alpha 版升级或在 Alpha 版之间升级。 + +### 2. 为 Rancher 创建命名空间 + +你需要定义一个 Kubernetes 命名空间，用于安装由 Chart 创建的资源。这个命名空间的名称为 `cattle-system`： + +``` +kubectl create namespace cattle-system +``` + +### 3. 选择 SSL 配置 + +Rancher Management Server 默认需要 SSL/TLS 配置来保证访问的安全性。 + +:::note + +如果你想在外部终止 SSL/TLS，请参见[外部负载均衡器的 TLS 终止](../getting-started/installation-and-upgrade/installation-references/helm-chart-options.md#外部-tls-终止)。 + +::: + +你可以从以下三种证书来源中选择一种，用于在 Rancher Server 中终止 TLS： + +- **Rancher 生成的 TLS 证书**：要求你在集群中安装 `cert-manager`。Rancher 使用 `cert-manager` 签发并维护证书。Rancher 会生成自己的 CA 证书，并使用该 CA 签署证书。然后 `cert-manager`负责管理该证书。 +- **Let's Encrypt**：Let's Encrypt 选项也需要使用 `cert-manager`。但是，在这种情况下，cert-manager 与 Let's Encrypt 的特殊颁发者相结合，该颁发者执行获取 Let's Encrypt 颁发的证书所需的所有操作（包括请求和验证）。此配置使用 HTTP 验证（`HTTP-01`），因此负载均衡器必须具有可以从互联网访问的公共 DNS 记录。 +- **你已有的证书**：使用已有的 CA 颁发的公有或私有证书。Rancher 将使用该证书来保护 WebSocket 和 HTTPS 流量。在这种情况下，你必须上传名称分别为 `tls.crt` 和 `tls.key`的 PEM 格式的证书以及相关的密钥。如果你使用私有 CA，则还必须上传该 CA 证书。这是由于你的节点可能不信任此私有 CA。Rancher 将获取该 CA 证书，并从中生成一个校验和，各种 Rancher 组件将使用该校验和来验证其与 Rancher 的连接。 + + +| 配置 | Helm Chart 选项 | 是否需要 cert-manager | +| ------------------------------ | ----------------------- | ------------------------------------- | +| Rancher 生成的证书（默认） | `ingress.tls.source=rancher` | [是](#4-安装-cert-manager) | +| Let’s Encrypt | `ingress.tls.source=letsEncrypt` | [是](#4-安装-cert-manager) | +| 你已有的证书 | `ingress.tls.source=secret` | 否 | + +### 4. 安装 cert-manager + +> 如果你使用自己的证书文件（`ingress.tls.source=secret`）或使用[外部负载均衡器的 TLS 终止](../installation-references/helm-chart-options.md#外部-tls-终止)，你可以跳过此步骤。 + +仅在使用 Rancher 生成的证书（`ingress.tls.source=rancher`）或 Let's Encrypt 颁发的证书（`ingress.tls.source=letsEncrypt`）时，才需要安装 cert-manager。 + +
+ 单击展开 + +:::note 重要提示： + +由于 cert-manager 的最新改动，你需要升级 cert-manager 版本。如果你需要升级 Rancher 并使用低于 0.11.0 的 cert-manager 版本，请参见[升级文档](../resources/upgrade-cert-manager.md)。 + +::: + +这些说明来自 [cert-manager 官方文档](https://cert-manager.io/docs/installation/kubernetes/#installing-with-helm)。 + +:::note + +要查看自定义 cert-manager 安装的选项（包括集群使用 PodSecurityPolicies 的情况），请参阅 [cert-manager 文档](https://artifacthub.io/packages/helm/cert-manager/cert-manager#configuration)。 + +::: + +``` +# 如果你手动安装了CRD，而不是在 Helm 安装命令中添加了 `--set installCRDs=true` 选项，你应该在升级 Helm Chart 之前升级 CRD 资源。 +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download//cert-manager.crds.yaml + +# 添加 Jetstack Helm 仓库 +helm repo add jetstack https://charts.jetstack.io + +# 更新本地 Helm Chart 仓库缓存 +helm repo update + +# 安装 cert-manager Helm Chart +helm install cert-manager jetstack/cert-manager \ + --namespace cert-manager \ + --create-namespace +``` + +安装完 cert-manager 后，你可以通过检查 cert-manager 命名空间中正在运行的 Pod 来验证它是否已正确部署： + +``` +kubectl get pods --namespace cert-manager + +NAME READY STATUS RESTARTS AGE +cert-manager-5c6866597-zw7kh 1/1 Running 0 2m +cert-manager-cainjector-577f6d9fd7-tr77l 1/1 Running 0 2m +cert-manager-webhook-787858fcdb-nlzsq 1/1 Running 0 2m +``` + +
+ +### 5. 根据你选择的证书选项，通过 Helm 安装 Rancher + +不同的证书配置需要使用不同的 Rancher 安装命令。 + +但是，无论证书如何配置，Rancher 在 `cattle-system` 命名空间中的安装名称应该总是 `rancher`。 + +:::tip 测试和开发： + +这个安装 Rancher 的最终命令需要一个将流量转发到 Rancher 的域名。如果你使用 Helm CLI 设置概念证明，则可以在传入 `hostname` 选项时使用伪域名。伪域名的一个例子是 `.sslip.io`，这会把 Rancher 暴露在它运行的 IP 上。生产安装中要求填写真实的域名。 + +::: + + + + +默认情况是使用 Rancher 生成 CA，并使用 `cert-manager` 颁发用于访问 Rancher Server 接口的证书。 + +由于 `rancher` 是 `ingress.tls.source` 的默认选项，因此在执行 `helm install` 命令时，我们不需要指定 `ingress.tls.source`。 + +- 将 `hostname` 设置为解析到你的负载均衡器的 DNS 名称。 +- 将 `bootstrapPassword` 设置为 `admin` 用户独有的值。 +- 如果你需要安装指定的 Rancher 版本，使用 `--version` 标志，例如 `--version 2.7.0`。 +- 对于 Kubernetes v1.25 或更高版本，使用 Rancher v2.7.2-v2.7.4 时，将 `global.cattle.psp.enabled` 设置为 `false`。对于 Rancher v2.7.5 及更高版本来说，这不是必需的，但你仍然可以手动设置该选项。 + +``` +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set bootstrapPassword=admin +``` + +如果你安装的是 alpha 版本，Helm 会要求你在安装命令中添加 `--devel` 选项： + +``` +helm install rancher rancher-alpha/rancher --devel +``` + +等待 Rancher 运行： + +``` +kubectl -n cattle-system rollout status deploy/rancher +Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... +deployment "rancher" successfully rolled out +``` + + + + +此选项使用 `cert-manager` 来自动请求和续订 [Let's Encrypt](https://letsencrypt.org/) 证书。Let's Encrypt 是免费的，而且是受信的 CA，因此可以为你提供有效的证书。 + +:::note + +由于 HTTP-01 质询只能在端口 80 上完成，因此你需要打开端口 80。 + +::: + +在以下命令中， + +- 将 `hostname` 设置为公有 DNS 记录。 +- 将 `bootstrapPassword` 设置为 `admin` 用户独有的值。 +- 将 `ingress.tls.source` 设置为 `letsEncrypt`。 +- 将 `letsEncrypt.email` 设置为可通讯的电子邮件地址，用于发送通知（例如证书到期的通知）。 +- 将 `letsEncrypt.ingress.class` 设为你的 Ingress Controller（例如 `traefik`，`nginx`，`haproxy`） +- 对于 Kubernetes v1.25 或更高版本，使用 Rancher v2.7.2-v2.7.4 时，将 `global.cattle.psp.enabled` 设置为 `false`。对于 Rancher v2.7.5 及更高版本来说，这不是必需的，但你仍然可以手动设置该选项。 + +``` +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set bootstrapPassword=admin \ + --set ingress.tls.source=letsEncrypt \ + --set letsEncrypt.email=me@example.org \ + --set letsEncrypt.ingress.class=nginx +``` + +如果你安装的是 alpha 版本，Helm 会要求你在安装命令中添加 `--devel` 选项： + +``` +helm install rancher rancher-alpha/rancher --devel +``` + +等待 Rancher 运行： + +``` +kubectl -n cattle-system rollout status deploy/rancher +Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... +deployment "rancher" successfully rolled out +``` + + + +在此选项中，你使用你自己的证书来创建 Kubernetes 密文，以供 Rancher 使用。 + +运行这个命令时，`hostname` 选项必须与服务器证书中的 `Common Name` 或 `Subject Alternative Names` 条目匹配，否则 Ingress controller 将无法正确配置。 + +虽然技术上仅需要 `Subject Alternative Names` 中有一个条目，但是拥有一个匹配的 `Common Name` 可以最大程度地提高与旧版浏览器/应用的兼容性。 + +:::note + +如果你想检查证书是否正确，请查看[如何在服务器证书中检查 Common Name 和 Subject Alternative Names](../../../faq/technical-items.md#如何在服务器证书中检查-common-name-和-subject-alternative-names)。 + +::: + +- 设置 `hostname`。 +- 将 `bootstrapPassword` 设置为 `admin` 用户独有的值。 +- 将 `ingress.tls.source` 设置为 `secret`。 +- 对于 Kubernetes v1.25 或更高版本，使用 Rancher v2.7.2-v2.7.4 时，将 `global.cattle.psp.enabled` 设置为 `false`。对于 Rancher v2.7.5 及更高版本来说，这不是必需的，但你仍然可以手动设置该选项。 + +``` +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set bootstrapPassword=admin \ + --set ingress.tls.source=secret +``` +如果你安装的是 alpha 版本，Helm 会要求你在安装命令中添加 `--devel` 选项： + +``` +helm install rancher rancher-alpha/rancher --devel +``` + +如果你使用的是私有 CA 证书，请在命令中增加 `--set privateCA=true`。 + +``` +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set bootstrapPassword=admin \ + --set ingress.tls.source=secret \ + --set privateCA=true +``` + +**添加 TLS 密文（千万不要遗漏此步骤）**：现在 Rancher 已经完成部署，你还需要参考[添加 TLS 密文](../resources/add-tls-secrets.md)发布证书文件，以便 Rancher 和 Ingress Controller 可以使用它们。 + + + + +Rancher Chart 有许多选项，用于为你的具体环境自定义安装。以下是一些常见的高级方案： + +- [HTTP 代理](../installation-references/helm-chart-options.md#http-代理) +- [私有容器镜像仓库](../installation-references/helm-chart-options.md#私有仓库和离线安装) +- [外部负载均衡器上的 TLS 终止](../installation-references/helm-chart-options.md#外部-tls-终止) + +如需获取完整的选项列表，请参见 [Chart 选项](../installation-references/helm-chart-options.md)。 + + +### 6. 验证 Rancher Server 是否部署成功 + +添加密文后，检查 Rancher 是否已成功运行： + +``` +kubectl -n cattle-system rollout status deploy/rancher +Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... +deployment "rancher" successfully rolled out +``` + +如果你看到 `error: deployment "rancher" exceeded its progress deadline` 这个错误，可运行以下命令来检查 deployment 的状态： + +``` +kubectl -n cattle-system get deploy rancher +NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE +rancher 3 3 3 3 3m +``` + +`DESIRED` 和 `AVAILABLE`的个数应该相同。 + +### 7. 保存选项 + +请保存你使用的 `--set` 选项。使用 Helm 升级 Rancher 到新版本时，你将需要使用相同的选项。 + +### 安装完成 + +安装已完成。现在 Rancher Server 应该已经可以正常运行了。 + +使用浏览器打开把流量转发到你的负载均衡器的 DNS 域名。然后，你就会看到一个漂亮的登录页面了。 + +如果遇到任何问题，请参见[故障排除](troubleshooting.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md index 22de9e7af35..4db511d97a0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md @@ -6,7 +6,7 @@ title: 在 Azure Kubernetes Service 上安装 Rancher 本指南使用命令行工具来配置一个带有 Ingress 的 AKS 集群。如果你更喜欢使用 Azure 门户来配置集群，请参见[官方文档](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-portal)。 -如果你已有一个 AKS Kubernetes 集群，请直接跳到[安装 Ingress](#5-安装-ingress) 的步骤，然后按照[此页](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。 +如果你已有一个 AKS Kubernetes 集群，请直接跳到[安装 Ingress](#5-安装-ingress) 的步骤，然后按照[此页](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。 ## 先决条件 @@ -19,7 +19,7 @@ title: 在 Azure Kubernetes Service 上安装 Rancher - [Microsoft Azure 账号](https://azure.microsoft.com/en-us/free/)：用于创建部署 Rancher 和 Kubernetes 的资源。 - [Microsoft Azure 订阅](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription#create-a-subscription-in-the-azure-portal)：如果你没有的话，请访问此链接查看如何创建 Microsoft Azure 订阅。 - [Micsoroft Azure 租户](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-create-new-tenant)：访问此链接并参考教程以创建 Microsoft Azure 租户。 -- 你的订阅有足够的配额，至少有 2 个 vCPU。有关 Rancher Server 资源要求的详情，请参见[此节](../../../pages-for-subheaders/installation-requirements.md#rke-和托管-kubernetes)。 +- 你的订阅有足够的配额，至少有 2 个 vCPU。有关 Rancher Server 资源要求的详情，请参见[此节](../installation-requirements/installation-requirements.md)。 - 在 Azure 中用 Helm 安装 Rancher 时，请使用 L7 负载均衡器来避免网络问题。详情请参见 [Azure 负载均衡器限制](https://docs.microsoft.com/en-us/azure/load-balancer/components#limitations)。 ## 1. 准备你的工作站 @@ -134,7 +134,7 @@ ingress-nginx-controller LoadBalancer 10.0.116.18 40.31.180.83 80:31229 ## 8. 安装 Rancher Helm Chart -按照[本页](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。任何 Kubernetes 发行版上安装的 Rancher 的 Helm 说明都是一样的。 +按照[本页](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。任何 Kubernetes 发行版上安装的 Rancher 的 Helm 说明都是一样的。 安装 Rancher 时，使用上一步获取的 DNS 名称作为 Rancher Server 的 URL。它可以作为 Helm 选项传递进来。例如，如果 DNS 名称是 `rancher.my.org`，你需要使用 `--set hostname=rancher.my.org` 选项来运行 Helm 安装命令。 @@ -144,4 +144,4 @@ ingress-nginx-controller LoadBalancer 10.0.116.18 40.31.180.83 80:31229 --set ingress.ingressClassName=nginx ``` -请参阅[Helm 安装命令](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 +请参阅[Helm 安装命令](install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md index 335e3fee425..efc95fe7e13 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md @@ -4,7 +4,7 @@ title: 在 Amazon EKS 上安装 Rancher 本文介绍了如何在 Amazon EKS 集群上安装 Rancher。你也可以[通过 AWS Marketplace 安装 Rancher](../../quick-start-guides/deploy-rancher-manager/aws-marketplace.md)。 -如果你已经有一个 EKS Kubernetes 集群，请直接跳转到[安装 Ingress](#5-安装-ingress)这个步骤。然后按照[此处](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的步骤安装 Rancher Helm Chart。 +如果你已经有一个 EKS Kubernetes 集群，请直接跳转到[安装 Ingress](#5-安装-ingress)这个步骤。然后按照[此处](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的步骤安装 Rancher Helm Chart。 ## 为 Rancher Server 创建 EKS 集群 @@ -138,7 +138,7 @@ ingress-nginx-controller LoadBalancer 10.100.90.18 a904a952c73bf4f668a17c4 ### 8. 安装 Rancher Helm Chart -按照[本页](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。任何 Kubernetes 发行版上安装的 Rancher 的 Helm 说明都是一样的。 +按照[本页](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。任何 Kubernetes 发行版上安装的 Rancher 的 Helm 说明都是一样的。 安装 Rancher 时，使用上一步获取的 DNS 名称作为 Rancher Server 的 URL。它可以作为 Helm 选项传递进来。例如，如果 DNS 名称是 `rancher.my.org`，你需要使用 `--set hostname=rancher.my.org` 选项来运行 Helm 安装命令。 @@ -148,4 +148,4 @@ ingress-nginx-controller LoadBalancer 10.100.90.18 a904a952c73bf4f668a17c4 --set ingress.ingressClassName=nginx ``` -请参阅[Helm 安装命令](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 +请参阅[Helm 安装命令](install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md index 90d395c0f12..04c2d4ff181 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md @@ -4,13 +4,13 @@ title: 在 GKE 集群上安装 Rancher 在本节中，你将学习如何使用 GKE 安装 Rancher。 -如果你已经有一个 GKE Kubernetes 集群，请直接跳转到[安装 Ingress](#7-安装-ingress)这个步骤。然后按照[此处](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的步骤安装 Rancher Helm Chart。 +如果你已经有一个 GKE Kubernetes 集群，请直接跳转到[安装 Ingress](#7-安装-ingress)这个步骤。然后按照[此处](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的步骤安装 Rancher Helm Chart。 ## 先决条件 - 你需要有一个 Google 账号。 - 你需要有一个 Google Cloud Billing 账号。你可使用 Google Cloud Console 来管理你的 Cloud Billing 账号。有关 Cloud Console 的详情，请参见 [ Console 通用指南](https://support.google.com/cloud/answer/3465889?hl=en&ref_topic=3340599)。 -- 你需要至少一个在用的 IP 地址和至少 2 个 CPU 的云配额。有关 Rancher Server 的硬件要求，请参见[本节](../../../pages-for-subheaders/installation-requirements.md#rke-和托管-kubernetes)。 +- 你需要至少一个在用的 IP 地址和至少 2 个 CPU 的云配额。有关 Rancher Server 的硬件要求，请参见[本节](../installation-requirements/installation-requirements.md)。 ## 1. 启用 Kubernetes Engine API @@ -190,7 +190,7 @@ ingress-nginx-controller LoadBalancer 10.3.244.156 35.233.206.34 80:3187 --set ingress.ingressClassName=nginx ``` -请参阅[Helm 安装命令](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 +请参阅[Helm 安装命令](install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 在 Rancher v2.7.5 中，如果你打算在集群上使用默认的 GKE Ingress 而不启用 VPC 原生的集群模式，则需要设置以下标志： diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades.md index 9e5367abb2d..b60d3ab748b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades.md @@ -28,11 +28,13 @@ kubeconfig 也可以通过 `--kubeconfig` 标签（详情请参见 https://helm. 本安装指南假定你使用的是 Helm 3。 + + 如果你使用 Helm 2，请参见 [Helm 2 迁移到 Helm 3 文档](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/)。如果你不能升级到 Helm 3，[Helm 2 升级页面](/versioned_docs/version-2.0-2.4/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades/helm2.md)提供了使用 Helm 2 升级的旧升级指南。 ### 离线安装：推送镜像到私有镜像仓库 -[仅适用于离线安装](../../../pages-for-subheaders/air-gapped-helm-cli-install.md)：为新的 Rancher Server 版本收集和推送镜像。使用你需要针对 Rancher 版本升级的镜像，按照步骤[推送镜像到私有镜像仓库](../other-installation-methods/air-gapped-helm-cli-install/publish-images.md)。 +[仅适用于离线安装](../other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)：为新的 Rancher Server 版本收集和推送镜像。使用你需要针对 Rancher 版本升级的镜像，按照步骤[推送镜像到私有镜像仓库](../other-installation-methods/air-gapped-helm-cli-install/publish-images.md)。 ### 使用 cert-manager 0.8.0 之前的版本升级 @@ -59,7 +61,7 @@ kubeconfig 也可以通过 `--kubeconfig` 标签（详情请参见 https://helm. 1. 获取你用来安装 Rancher 的仓库名称。 - 关于仓库及其区别，请参见 [Helm Chart Repositories](../resources/choose-a-rancher-version.md#helm-chart-仓库)。 + 关于仓库及其区别，请参见 [Helm Chart 仓库](../resources/choose-a-rancher-version.md#helm-chart-仓库)。 - Latest：建议用于试用最新功能 ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-and-upgrade.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-and-upgrade.md new file mode 100644 index 00000000000..aeef9f9c6e7 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-and-upgrade.md @@ -0,0 +1,94 @@ +--- +title: 安装/升级 Rancher +description: 了解如何在开发和生产环境中安装 Rancher。了解单节点和高可用安装 +--- + +本节介绍了 Rancher 各种安装方式以及每个安装方式的优点。 + +## 名词解释 + +本章节涉及以下名词： + +- **Rancher Server**：用于管理和配置 Kubernetes 集群。你可以通过 Rancher Server 的 UI 与下游 Kubernetes 集群进行交互。Rancher Management Server 可以安装到任意 Kubernetes 集群上，包括托管的集群，如 Amazon EKS 集群。 +- **RKE（Rancher Kubernetes Engine）**：是经过认证的 Kubernetes 发行版，也是用于创建和管理 Kubernetes 集群的 CLI 工具和库。 +- **K3s（轻量级 Kubernetes）**：也是经过认证的 Kubernetes 发行版。它比 RKE 更新，更易用且更轻量，其所有组件都在一个小于 100 MB 的二进制文件中。 +- **RKE2**：一个完全合规的 Kubernetes 发行版，专注于安全和合规性。 + +`restrictedAdmin` Helm Chart 选项在 **Rancher Server** 可用。如果该选项设置为 true，初始的 Rancher 用户访问本地 Kubernetes 集群会受到限制，以避免权限升级。详情请参见 [restricted-admin 角色](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md#受限管理员)。 + +## 安装方式概述 + +Rancher 可以安装在以下主要架构上： + +### 使用 Helm CLI 安装的高可用 Kubernetes + +我们建议使用 Kubernetes 包管理器 Helm 在专用的 Kubernetes 集群上安装 Rancher。在 RKE 集群中，需要使用三个节点才能实现高可用集群。在 K3s 集群中，只需要两个节点即可。 + +### 通过 AWS Marketplace 在 EKS 上安装 Rancher + +你可以[通过 AWS Marketplace](../quick-start-guides/deploy-rancher-manager/aws-marketplace.md) 将 Rancher 安装到 Amazon Elastic Kubernetes Service (EKS) 上。部署的 EKS 集群已生产就绪，并遵循 AWS 最佳实践。 + +### 单节点 Kubernetes 安装 + +Rancher 可以安装在单节点 Kubernetes 集群上。但是，在单节点安装的情况下，Rancher Server 没有高可用性。而高可用性对在生产环境中运行 Rancher 非常重要。 + +但是，如果你想要短期内使用单节点节省资源，同时又保留高可用性迁移路径，那么单节点 Kubernetes 安装也是合适的。你也可以之后向集群中添加节点，获得高可用的 Rancher Server。 + +### Docker 安装 + +如果你的目的是测试或演示，你可以使用 Docker 把 Rancher 安装到单个节点中。本地 Kubernetes 集群是安装到单个 Docker 容器中的，而 Rancher 是安装到本地集群中的。 + +Rancher backup operator 可将 Rancher 从单个 Docker 容器迁移到高可用 Kubernetes 集群上。详情请参见[把 Rancher 迁移到新集群](../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)。 + +### 其他方式 + +如果你需要在离线环境中或使用 HTTP 代理安装 Rancher，请参见以下独立的说明文档： + +| 网络访问方式 | 基于 Kubernetes 安装（推荐） | 基于 Docker 安装 | +| ---------------------------------- | ------------------------------ | ---------- | +| 可直接访问互联网 | [文档](install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md) | [文档](other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md) | +| 使用 HTTP 代理 | [文档](other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md) | [文档](other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md)及[配置](../../reference-guides/single-node-rancher-in-docker/http-proxy-configuration.md) | +| 离线环境 | [文档](other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md) | [文档](other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md) | + +我们建议在 Kubernetes 集群上安装 Rancher，因为在多节点集群中，Rancher Server 可以实现高可用。高可用配置可以提升 Rancher 访问其管理的下游 Kubernetes 集群的稳定性。 + +因此，我们建议在生产级别的架构中，设置一个高可用的 Kubernetes 集群，然后在这个集群上安装 Rancher。安装 Rancher 后，你可以使用 Rancher 部署和管理 Kubernetes 集群。 + +如果你的目的是测试或演示，你可以将 Rancher 安装到单个 Docker 容器中。Docker 安装可以让你实现开箱即用，以使用 Rancher 设置 Kubernetes 集群。Docker 安装主要是用于探索 Rancher Server 的功能，只适用于开发和测试。 + +[在 Kubernetes 上安装 Rancher 的说明](install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)介绍了如何首先使用 K3s 或 RKE 创建和管理 Kubernetes 集群，然后再将 Rancher 安装到该集群上。 + +如果 Kubernetes 集群中的节点正在运行且满足[节点要求](installation-requirements/installation-requirements.md)，你可以使用 Helm 将 Rancher 部署到 Kubernetes 上。Helm 使用 Rancher 的 Helm Chart 在 Kubernetes 集群的每个节点上安装 Rancher 的副本。我们建议使用负载均衡器将流量定向到集群中的每个 Rancher 副本上。 + +如需进一步了解 Rancher 架构，请参见[架构概述](../../reference-guides/rancher-manager-architecture/rancher-manager-architecture.md)，[生产级别架构推荐](../../reference-guides/rancher-manager-architecture/architecture-recommendations.md)或[最佳实践指南](../../reference-guides/best-practices/rancher-server/tips-for-running-rancher.md)。 + +## 先决条件 + +安装 Rancher 之前，请确保你的节点满足所有[安装要求](installation-requirements/installation-requirements.md)。 + +## 架构建议 + +为了达到最佳性能和安全性，我们建议你为 Rancher Management Server 使用单独的专用 Kubernetes 集群。不建议在此集群上运行用户工作负载。部署 Rancher 后，你可以[创建或导入集群](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)来运行你的工作负载。 + +详情请参见[架构推荐](../../reference-guides/rancher-manager-architecture/architecture-recommendations.md)。 + +### 在 Kubernetes 上安装 Rancher 的更多选项 + +参见 [Helm Chart 选项](installation-references/helm-chart-options.md)以了解在 Kubernetes 集群上安装 Rancher 的其他配置，包括： + +- [开启 API 审计日志来记录所有事务](installation-references/helm-chart-options.md#api-审计日志) +- [负载均衡器上的 TLS 终止](installation-references/helm-chart-options.md#外部-tls-终止) +- [自定义 Ingress](installation-references/helm-chart-options.md#自定义-ingress) + +在 Rancher 的安装指南中，我们推荐使用 K3s 或 RKE 来配置 Kubernetes 集群，然后再在这个集群中安装 Rancher。K3s 和 RKE 均提供许多配置选项，用于为你的具体环境自定义 Kubernetes 集群。有关选项和功能的完整列表，请参见： + +- [RKE 配置选项](https://rancher.com/docs/rke/latest/en/config-options/) +- [K3s 配置选项](https://rancher.com/docs/k3s/latest/en/installation/install-options/) + +### 在 Docker 上安装 Rancher 的更多选项 + +参见 [Docker 安装选项](other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md)了解其他配置，包括： + +- [开启 API 审计日志来记录所有事务](../../reference-guides/single-node-rancher-in-docker/advanced-options.md#api-审计日志) +- [外部负载均衡器](../../how-to-guides/advanced-user-guides/configure-layer-7-nginx-load-balancer.md) +- [持久化数据存储](../../reference-guides/single-node-rancher-in-docker/advanced-options.md#持久化数据) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/feature-flags.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/feature-flags.md index 873a0689b14..5a022b13135 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/feature-flags.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/feature-flags.md @@ -4,7 +4,7 @@ title: 功能开关 使用功能开关（Feature Flag），你可以试用可选或实验性的功能并启用正在逐步淘汰的旧版功能。 -要了解功能的值以及如何启用它们，请参阅[启用实验性功能](../../../pages-for-subheaders/enable-experimental-features.md)。 +要了解功能的值以及如何启用它们，请参阅[启用实验性功能](../../../how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 :::note @@ -16,7 +16,7 @@ title: 功能开关 - `continuous-delivery`：允许从 Fleet 中单独禁用 Fleet GitOps。有关详细信息，请参阅[持续交付](../../../how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md)。 - `fleet`：v2.6 及更高版本的 Rancher 配置框架需要 Fleet。即使你在旧 Rancher 版本中禁用了该标志，该标志也将在升级时自动启用。有关详细信息，请参阅 [Fleet - GitOps at Scale](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md)。 -- `harvester`：管理 Virtualization Management 页面的访问。用户可以在该页面直接导航到 Harvester 集群并访问 Harvester UI。有关详细信息，请参阅 [Harvester 集成](../../../integrations-in-rancher/harvester.md)。 +- `harvester`：管理 Virtualization Management 页面的访问。用户可以在该页面直接导航到 Harvester 集群并访问 Harvester UI。有关详细信息，请参阅 [Harvester 集成](../../../integrations-in-rancher/harvester/overview.md)。 - `istio-virtual-service-ui`：启用[可视界面](../../../how-to-guides/advanced-user-guides/enable-experimental-features/istio-traffic-management-features.md)来创建、读取、更新和删除 Istio 虚拟服务和目标规则，这些都是 Istio 流量管理功能。 - `legacy`：启用 2.5.x 及更早版本的一组功能，这些功能正逐渐被新的实现淘汰。它们是已弃用以及后续可用于新版本的功能组合。新的 Rancher 安装会默认禁用此标志。如果你从以前版本的 Rancher 升级，此标志会启用。 - `multi-cluster-management`：允许配置和管理多个 Kubernetes 集群。此标志只能在安装时设置。后续无法启用或禁用它。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index bd69a89cfee..c924cc3c797 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -7,7 +7,7 @@ keywords: [rancher helm chart, rancher helm 选项, rancher helm chart 选项, h 如需选择 Helm Chart 版本，请参见[本页](../../../getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md)。 -了解开启实验性功能的详情，请参见[本页](../../../pages-for-subheaders/enable-experimental-features.md)。 +了解开启实验性功能的详情，请参见[本页](../../../how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 ## 常用选项 @@ -81,13 +81,13 @@ kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ 启用 [API 审计日志](../../../how-to-guides/advanced-user-guides/enable-api-audit-log.md)。 -你可以像收集其他容器日志一样收集此日志。在 Rancher Server 集群上为 `System` 项目启用 [Logging](../../../pages-for-subheaders/logging.md)。 +你可以像收集其他容器日志一样收集此日志。在 Rancher Server 集群上为 `System` 项目启用 [Logging](../../../integrations-in-rancher/logging/logging.md)。 ```plain --set auditLog.level=1 ``` -默认情况下，启用审计日志会在 Rancher pod 中创建一个 Sidecar 容器。这个容器（`rancher-audit-log`）会把日志流传输到 `stdout`。你可以像收集其他容器日志一样收集此日志。如果你使用 Sidecar 作为审计日志的目标时， `hostPath`，`maxAge`，`maxBackups` 和 `maxSize` 选项不会生效。建议使用你的操作系统或 Docker Daemon 的日志轮换功能来控制磁盘空间的使用。请为 Rancher Server 集群或 System 项目启用 [Logging](../../../pages-for-subheaders/logging.md)。 +默认情况下，启用审计日志会在 Rancher pod 中创建一个 Sidecar 容器。这个容器（`rancher-audit-log`）会把日志流传输到 `stdout`。你可以像收集其他容器日志一样收集此日志。如果你使用 Sidecar 作为审计日志的目标时， `hostPath`，`maxAge`，`maxBackups` 和 `maxSize` 选项不会生效。建议使用你的操作系统或 Docker Daemon 的日志轮换功能来控制磁盘空间的使用。请为 Rancher Server 集群或 System 项目启用 [Logging](../../../integrations-in-rancher/logging/logging.md)。 将 `auditLog.destination` 的值设为 `hostPath`，可以将日志转发到与主机系统共享的卷，而不是传输到 Sidecar 容器。如果目标设置为 `hostPath`，你可能需要调整其他 auditLog 参数以进行日志轮换。 @@ -202,7 +202,7 @@ kubectl -n cattle-system create secret generic tls-ca-additional --from-file=ca- ### 私有仓库和离线安装 -有关使用私有仓库安装 Rancher 的详情，请参见[离线安装](../../../pages-for-subheaders/air-gapped-helm-cli-install.md)。 +有关使用私有仓库安装 Rancher 的详情，请参见[离线安装](../other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)。 ## 外部 TLS 终止 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/installation-references.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/installation-references.md new file mode 100644 index 00000000000..c2d9035c3cd --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/installation-references.md @@ -0,0 +1,5 @@ +--- +title: 安装参考 +--- + +有关其他安装资源，请参阅以下参考指南：[Rancher Helm Chart 选项](helm-chart-options.md)、[TLS 设置](tls-settings.md)和[功能开关](feature-flags.md)。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-requirements/install-docker.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-requirements/install-docker.md index 8b98ce09030..27bf949bb11 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-requirements/install-docker.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-requirements/install-docker.md @@ -2,22 +2,22 @@ title: 安装 Docker --- -在使用 Helm 或 Docker 在 RKE 集群节点上安装 Rancher Server 前，你需要先安装 Docker。RKE2 和 K3s 集群不要求使用 Docker。 +在使用 Helm 在 RKE 集群节点上或使用 Docker 安装 Rancher Server 前，你需要在节点中先安装 Docker。RKE2 和 K3s 集群不要求使用 Docker。 Docker 有几个安装方法。一种方法是参见 [Docker 官方文档](https://docs.docker.com/install/)以了解如何在 Linux 上安装 Docker。不同 Linux 发行版的安装步骤可能有所不同。 -另一种方式是使用 Rancher 的 Docker 安装脚本，该脚本可用于较新的 Docker 版本。 +另一种方式是使用 Rancher 的 Docker 安装脚本，该脚本可用于较新的 Docker 版本。 Rancher 为每个 Kubernetes 支持的上游 Docker 版本提供了安装脚本。 -例如，此命令可用于在 SUSE Linux Enterprise 或 Ubuntu 等主要 Linux 发行版上安装 Docker 20.10： +例如，此命令可用于在 SUSE Linux Enterprise 或 Ubuntu 等主要 Linux 发行版上安装 Docker ： -``` -curl https://releases.rancher.com/install-docker/20.10.sh | sh +```bash +curl https://releases.rancher.com/install-docker/.sh | sh ``` -Rancher 提供 Kubernetes 支持的所有上游 Docker 版本的安装脚本。如需了解我们是否提供某个 Docker 版本的安装脚本，请参见包含了 Rancher 所有的 Docker 安装脚本的 [GitHub 仓库](https://github.com/rancher/install-docker)。 +请参阅 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix)，使用匹配你的操作系统和 Rancher 版本并且经过验证的 Docker 版本。 尽管支持矩阵列出了经过验证的 Docker 版本直至补丁版本，但只有发行版的主要版本和次要版本与 Docker 安装脚本相关。 请注意，必须应用以下 sysctl 设置： -``` +```bash net.bridge.bridge-nf-call-iptables=1 ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md new file mode 100644 index 00000000000..4e9ec43a2a1 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md @@ -0,0 +1,215 @@ +--- +title: 安装要求 +description: 如果 Rancher 配置在 Docker 或 Kubernetes 中运行时，了解运行 Rancher Server 的每个节点的节点要求 +--- + +本文描述了对需要安装 Rancher Server 的节点的软件、硬件和网络要求。Rancher Server 可以安装在单个节点或高可用的 Kubernetes 集群上。 + +:::note 重要提示： + +如果你需要在 Kubernetes 集群上安装 Rancher，该节点的要求与用于运行应用和服务的[下游集群的节点要求](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)不同。 + +::: + +Rancher UI 在基于 Firefox 或 Chromium 的浏览器（Chrome、Edge、Opera、Brave）中效果最佳。 + +查看我们的[最佳实践](../../../reference-guides/best-practices/rancher-server/tips-for-running-rancher.md)页面，获取在生产环境中运行 Rancher Server 的建议。 + +## Kubernetes 与 Rancher 的兼容性 + +Rancher 需要安装在支持的 Kubernetes 版本上。请查阅 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions)，确保你的 Kubernetes 版本受支持。 + +## 操作系统和容器运行时要求 + +所有支持的操作系统都使用 64-bit x86 架构。Rancher 兼容当前所有的主流 Linux 发行版。 + +[Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions)列出了每个 Rancher 版本测试过的操作系统和 Docker 版本。 + +运行 RKE 集群的节点需要安装 Docker。RKE2 或 K3s 集群不需要它。 + +请安装 `ntp`（Network Time Protocol），以防止在客户端和服务器之间由于时间不同步造成的证书验证错误。 + +某些 Linux 发行版的默认防火墙规则可能会阻止 Kubernetes 集群内的通信。从 Kubernetes v1.19 开始，你必须关闭 firewalld，因为它与 Kubernetes 网络插件冲突。 + +如果你不太想这样做的话，你可以查看[相关问题](https://github.com/rancher/rancher/issues/28840)中的建议。某些用户已能成功[使用 ACCEPT 策略 为 Pod CIDR 创建一个独立的 firewalld 区域](https://github.com/rancher/rancher/issues/28840#issuecomment-787404822)。 + +如果你需要在 ARM64 上使用 Rancher，请参见[在 ARM64（实验功能）上运行 Rancher](../../../how-to-guides/advanced-user-guides/enable-experimental-features/rancher-on-arm64.md)。 + +### RKE2 要求 + +对于容器运行时，RKE2 附带了自己的 containerd。RKE2 安装不需要 Docker。 + +如需了解 RKE2 通过了哪些操作系统版本的测试，请参见 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions)。 + + +### K3s 要求 + +对于容器运行时，K3s 默认附带了自己的 containerd。你也可以将 K3s 配置为使用已安装的 Docker 运行时。有关在 Docker 中使用 K3s 的更多信息，请参阅 [K3s 文档](https://docs.k3s.io/advanced#using-docker-as-the-container-runtime)。 + +Rancher 需要安装在支持的 Kubernetes 版本上。如需了解你使用的 Rancher 版本支持哪些 Kubernetes 版本，请参见 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions)。如需指定 K3s 版本，在运行 K3s 安装脚本时，使用 `INSTALL_K3S_VERSION` 环境变量。 + +如果你使用 **Raspbian Buster** 在 K3s 集群上安装 Rancher，请按照[这些步骤](https://rancher.com/docs/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster)切换到旧版 iptables。 + +如果你使用 Alpine Linux 的 K3s 集群上安装 Rancher，请按照[这些步骤](https://rancher.com/docs/k3s/latest/en/advanced/#additional-preparation-for-alpine-linux-setup)进行其他设置。 + +### RKE 要求 + +RKE 需要 Docker 容器运行时。支持的 Docker 版本请参见 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions) + +有关详细信息，请参阅[安装 Docker](install-docker.md)。 + +## 硬件要求 + +本节描述安装 Rancher Server 的节点的 CPU、内存和磁盘要求。硬件要求根据你的 Rancher 部署规模而定。 + +### 实际考虑 + +Rancher 的硬件占用空间取决于许多因素，包括： + + - 托管的基础设施规模 (例如： 节点数量，集群数量)。 + - 所需访问控制规则的复杂性（例如：RoleBinding 对象计数）。 + - 工作负载数量 (例如： Kubernetes 部署，Fleet 部署)。 + - 使用模式 (例如：主动使用的功能集合，使用频率，并发用户数量). + +由于存在许多可能随时间变化的影响因素，因此此处列出的要求为适合大多数用例的起点。 然而，你的用例可能有不同的要求。 若你需要对于特定场景的咨询，请[联系 Rancher]((https://rancher.com/contact/)) 以获得进一步指导。 + + +特别指出，本页面中的要求基于以下假设的环境提出，包括： + - 每种类型的 Kubernetes 资源数量小于 60,000 个。 + - 每个节点最多 120 个 Pod。 + - 上游（本地）集群中最多 200 个 CRD。 + - 下游集群中最多 100 个 CRD。 + - 最多 50 个 Fleet 部署。 + +更多的数量也是能够达到的，但需要更高的硬件要求。 如果你有超过 20,000 个相同类型的资源，通过 Rancher UI 加载整个列表的时间可能需要几秒钟。 + +:::note Evolution: + +Rancher 的代码库不断发展，用例不断变化，Rancher 积累的经验也在不断增长。 + +随着指导方针的准确性不断的提高并且变得更加具体，硬件要求也会发生变化。 + +如果你发现你的 Rancher 部署不再符合列出的建议，请[联系 Rancher](https://rancher.com/contact/) 进行重新评估。 + +::: + +### RKE2 Kubernetes + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，生产环境下的高可用安装最少需要 3 个节点。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | +| --------------- | -------- | --------- | ----- | ---- | +| 小 | 150 | 1500 | 4 | 16 GB | +| 中 | 300 | 3000 | 8 | 32 GB | +| 大 (*) | 500 | 5000 | 16 | 64 GB | +| 更大 (†) | (†) | (†) | (†) | (†) | + +(*)： 大规模的部署需要你[遵循最佳实践](../../../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md)以获得足够的性能。 + +(†)： 通过特别的硬件建议和调整能够实现更大的部署规模。 你可以[联系 Rancher](https://rancher.com/contact/) 进行定制评估。 + +有关 RKE2 一般要求的更多详细信息，请参见 [RKE2 文档](https://docs.rke2.io/install/requirements)。 + +### K3s Kubernetes + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，生产环境下的高可用安装最少需要 3 个节点。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | 外部数据库(*) | +| --------------- | ---------- | ------------ | -------| ---------| ------------------------- | +| Small | 150 | 1500 | 4 | 16 GB | 2 vCPUs, 8 GB + 1000 IOPS | +| Medium | 300 | 3000 | 8 | 32 GB | 4 vCPUs, 16 GB + 2000 IOPS | +| Large (†) | 500 | 5000 | 16 | 64 GB | 8 vCPUs, 32 GB + 4000 IOPS | + +(*)：外部数据库是指将 K3s 集群数据存储在[专用的外部主机](https://docs.k3s.io/datastore)上。 这是可选的。 具体要求取决于使用的外部数据库。 + +(†)：大规模的部署需要你[遵循最佳实践](../../../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md)以获得足够的性能。 + +有关 K3s 一般要求的更多详细信息，请参见 [K3s 文档](https://docs.k3s.io/installation/requirements)。 + +### 托管 Kubernetes + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，生产环境下的高可用安装最少需要 3 个节点。 + +这些要求适用于托管 Kubernetes 集群，例如 Amazon Elastic Kubernetes Service (EKS)、Azure Kubernetes Service (AKS) 或 Google Kubernetes Engine (GKE)。 它们不适用于 Rancher SaaS 解决方案，例如 [Rancher Prime Hosted](https://www.rancher.com/products/rancher)。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | +|-----------------------------|----------------------------|-------------------------|-------|-------| +| 小 | 150 | 1500 | 4 | 16 GB | +| 中 | 300 | 3000 | 8 | 32 GB | +| 大 (*) | 500 | 5000 | 16 | 64 GB | + +(*)：大规模的部署需要你[遵循最佳实践](../../../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md)以获得足够的性能。 + + +### RKE + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，生产环境下的高可用安装最少需要 3 个节点。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | +|-----------------------------|----------------------------|-------------------------|-------|-------| +| 小 | 150 | 1500 | 4 | 16 GB | +| 中 | 300 | 3000 | 8 | 32 GB | +| 大 (*) | 500 | 5000 | 16 | 64 GB | + +(*)： 大规模的部署需要你[遵循最佳实践](../../../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md)以获得足够的性能。 + +有关 RKE 一般要求的更多详细信息，请参见 [RKE 文档](https://rke.docs.rancher.com/os)。 + +### Docker + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，在 Docker 中安装 Rancher 仅适用于开发或测试目的。不建议在生产环境中使用。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | +|-----------------------------|----------------------------|-------------------------|-------|------| +| 小 | 5 | 50 | 1 | 4 GB | +| 中 | 15 | 200 | 2 | 8 GB | + +## Ingress + +安装 Rancher 的 Kubernetes 集群中的每个节点都应该运行一个 Ingress。 + +Ingress 需要部署为 DaemonSet 以确保负载均衡器能成功把流量转发到各个节点。 + +如果是 RKE，RKE2 和 K3s 安装，你不需要手动安装 Ingress，因为它是默认安装的。 + +对于托管的 Kubernetes 集群（EKS、GKE、AKS），你需要设置 Ingress。 + +- **Amazon EKS**：[在 Amazon EKS 上安装 Rancher 以及如何安装 Ingress 以访问 Rancher Server](../install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md)。 +- **AKS**：[使用 Azure Kubernetes 服务安装 Rancher 以及如何安装 Ingress 以访问 Rancher Server](../install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md)。 +- **GKE**：[使用 GKE 安装 Rancher 以及如何安装 Ingress 以访问 Rancher Server](../install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md)。 + +## 磁盘 + +etcd 在集群中的性能决定了 Rancher 的性能。因此，为了获得最佳速度，我们建议使用 SSD 磁盘来支持 Rancher 管理的 Kubernetes 集群。在云提供商上，你还需使用能获得最大 IOPS 的最小大小。在较大的集群中，请考虑使用专用存储设备存储 etcd 数据和 wal 目录。 + +## 网络要求 + +本节描述了安装 Rancher Server 的节点的网络要求。 + +:::caution + +如果包含 Rancher 的服务器带有 `X-Frame-Options=DENY` 标头，在升级旧版 UI 之后，Rancher UI 中的某些页面可能无法渲染。这是因为某些旧版页面在新 UI 中是以 iFrames 模式嵌入的。 + +::: + +### 节点 IP 地址 + +无论你是在单个节点还是高可用集群上安装 Rancher，每个节点都应配置一个静态 IP。如果使用 DHCP，则每个节点都应该有一个 DHCP 预留，以确保节点分配到相同的 IP 地址。 + +### 端口要求 + +为了确保能正常运行，Rancher 需要在 Rancher 节点和下游 Kubernetes 集群节点上开放一些端口。不同集群类型的 Rancher 和下游集群的所有必要端口，请参见[端口要求](port-requirements.md)。 + +## Dockershim 支持 + +有关 Dockershim 支持的详情，请参见[此页面](dockershim.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-requirements/port-requirements.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-requirements/port-requirements.md index 7dcdc48683c..0bf72d2e721 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-requirements/port-requirements.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-requirements/port-requirements.md @@ -178,9 +178,9 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点 下游 Kubernetes 集群用于运行你的应用和服务。本节介绍了哪些端口需要在下游集群的节点上打开，以便 Rancher 能够与它们进行通信。 -不同的下游集群的启动方式有不同的端口要求。下面的每个标签都列出了不同[集群类型](../../../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)所需打开的端口。 +不同的下游集群的启动方式有不同的端口要求。下面的每个标签都列出了不同[集群类型](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)所需打开的端口。 -下图描述了为每个[集群类型](../../../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)打开的端口。 +下图描述了为每个[集群类型](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)打开的端口。
Rancher 管理面板的端口要求
@@ -202,7 +202,7 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点
单击展开 -下表描述了节点在[云提供商](../../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md)中创建的情况下，[Rancher 启动 Kubernetes](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md) 的端口要求。 +下表描述了节点在[云提供商](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)中创建的情况下，[Rancher 启动 Kubernetes](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) 的端口要求。 :::note @@ -219,7 +219,7 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点
单击展开 -下表描述了使用[自定义节点](../../../pages-for-subheaders/use-existing-nodes.md)的情况下，[Rancher 启动 Kubernetes](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md) 的端口要求。 +下表描述了使用[自定义节点](../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)的情况下，[Rancher 启动 Kubernetes](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) 的端口要求。 @@ -230,7 +230,7 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点
单击展开 -下表描述了[托管集群](../../../pages-for-subheaders/set-up-clusters-from-hosted-kubernetes-providers.md)的端口要求。 +下表描述了[托管集群](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md)的端口要求。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md new file mode 100644 index 00000000000..4e36f893125 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md @@ -0,0 +1,31 @@ +--- +title: 离线 Helm CLI 安装 +--- + +本文介绍如何使用 Helm CLI 在离线环境中安装 Rancher Server。离线环境可以是 Rancher Server 离线安装、防火墙后面或代理后面。 + +Rancher 安装在 RKE Kubernetes 集群、K3s Kubernetes 集群，或单个 Docker 容器上对应的安装步骤会有所不同。 + +如需了解各个安装方式的更多信息，请参见[本页](../../installation-and-upgrade.md)。 + +在安装指导中，我们为不同的安装选项提供对应的 _选项卡_ 。 + +:::note 重要提示： + +如果你按照 Docker 安装指南安装 Rancher，你将没有把 Docker 安装转换为 Kubernetes 安装的升级途径。 + +::: + +## 安装概要 + +1. [设置基础设施和私有镜像仓库](infrastructure-private-registry.md) +2. [收集镜像到私有镜像仓库](publish-images.md) +3. [设置 Kubernetes 集群（如果你使用 Docker 安装，请跳过此步骤）](install-kubernetes.md) +4. [安装 Rancher](install-rancher-ha.md) + +## 升级 + +如需在离线环境中使用 Helm CLI 升级 Rancher，请按照[升级步骤](../../install-upgrade-on-a-kubernetes-cluster/upgrades.md)进行操作。 + +### 后续操作 +[准备节点](infrastructure-private-registry.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/infrastructure-private-registry.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/infrastructure-private-registry.md index e3e7be1a627..8978a639b0e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/infrastructure-private-registry.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/infrastructure-private-registry.md @@ -6,7 +6,7 @@ title: '1. 设置基础设施和私有镜像仓库' 离线环境是 Rancher Server 离线安装或安装在防火墙后面的环境。 -Rancher 安装在 K3s Kubernetes 集群、RKE Kubernetes 集群还是单个 Docker 容器上对应的基础设施设置会有所不同。如需了解各个安装方式的更多信息，请参见[本页](../../../../pages-for-subheaders/installation-and-upgrade.md)。 +Rancher 安装在 K3s Kubernetes 集群、RKE Kubernetes 集群还是单个 Docker 容器上对应的基础设施设置会有所不同。如需了解各个安装方式的更多信息，请参见[本页](../../installation-and-upgrade.md)。 Rancher 可以安装在任何 Kubernetes 集群上。为了阅读方便，我们在下文中仍提供了 RKE 和 K3s Kubernetes 基础设施教程。 @@ -25,7 +25,7 @@ Rancher 可以安装在任何 Kubernetes 集群上。为了阅读方便，我们 这些主机会断开互联网链接，但需要能与你的私有镜像仓库连接。 -请确保你的节点满足[操作系统，容器运行时，硬件和网络](../../../../pages-for-subheaders/installation-requirements.md)的常规要求。 +请确保你的节点满足[操作系统，容器运行时，硬件和网络](../../installation-requirements/installation-requirements.md)的常规要求。 如需获取配置 Linux 节点的示例，请参见[在 Amazon EC2 中配置节点](../../../../how-to-guides/new-user-guides/infrastructure-setup/nodes-in-amazon-ec2.md)的教程。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha.md index 0aa220051f7..d28b59ed745 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha.md @@ -102,14 +102,14 @@ helm repo update 从 [Helm Chart 仓库](https://artifacthub.io/packages/helm/cert-manager/cert-manager)中获取最新可用的 cert-manager Chart： ```plain -helm fetch jetstack/cert-manager --version v1.11.0 +helm fetch jetstack/cert-manager ``` ##### 3. 检索 Cert-Manager CRD 为 cert-manager 下载所需的 CRD 文件： ```plain -curl -L -o cert-manager-crd.yaml https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml +curl -L -o cert-manager-crd.yaml https://github.com/cert-manager/cert-manager/releases/download//cert-manager.crds.yaml ``` ### 4. 安装 Rancher @@ -146,7 +146,7 @@ curl -L -o cert-manager-crd.yaml https://github.com/cert-manager/cert-manager/re 3. 安装 cert-manager。 ```plain - helm install cert-manager ./cert-manager-v1.11.0.tgz \ + helm install cert-manager ./cert-manager-.tgz \ --namespace cert-manager \ --set image.repository=/quay.io/jetstack/cert-manager-controller \ --set webhook.image.repository=/quay.io/jetstack/cert-manager-webhook \ diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md index c64f7132d1d..783555ba158 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md @@ -4,11 +4,11 @@ title: '2. 收集镜像并发布到私有仓库' 本文介绍如何配置私有镜像仓库，以便在安装 Rancher 时，Rancher 可以从此私有镜像仓库中拉取所需的镜像。 -默认情况下，所有用于[配置 Kubernetes 集群](../../../../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)或启动 Rancher 中的工具（如监控，流水线，告警等）的镜像都是从 Docker Hub 中拉取的。在 Rancher 的离线安装中，你需要一个私有仓库，该仓库位于你的 Rancher Server 中某个可访问的位置。然后，你可加载该存有所有镜像的镜像仓库。 +默认情况下，所有用于[配置 Kubernetes 集群](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)或启动 Rancher 中的工具（如监控，流水线，告警等）的镜像都是从 Docker Hub 中拉取的。在 Rancher 的离线安装中，你需要一个私有仓库，该仓库位于你的 Rancher Server 中某个可访问的位置。然后，你可加载该存有所有镜像的镜像仓库。 使用 Docker 安装 Rancher，和把 Rancher 安装到 Kubernetes 集群，其对应的推送镜像到私有镜像仓库步骤是一样的。 -你使用 Rancher 配置的下游集群是否有运行 Windows 的节点，决定了本文涉及的步骤。我们提供的推送镜像到私有镜像仓库步骤，是基于假设 Rancher 仅配置运行 Linux 节点的下游 Kubernetes 集群的。但是，如果你计划[在下游 Kubernetes 集群中使用 Windows 节点](../../../../pages-for-subheaders/use-windows-clusters.md)，我们有单独的文档来介绍如何为需要的镜像提供支持。 +你使用 Rancher 配置的下游集群是否有运行 Windows 的节点，决定了本文涉及的步骤。我们提供的推送镜像到私有镜像仓库步骤，是基于假设 Rancher 仅配置运行 Linux 节点的下游 Kubernetes 集群的。但是，如果你计划[在下游 Kubernetes 集群中使用 Windows 节点](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)，我们有单独的文档来介绍如何为需要的镜像提供支持。 :::note 先决条件： @@ -67,7 +67,7 @@ title: '2. 收集镜像并发布到私有仓库' ```plain helm repo add jetstack https://charts.jetstack.io helm repo update - helm fetch jetstack/cert-manager --version v1.11.0 + helm fetch jetstack/cert-manager helm template ./cert-manager-.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt ``` @@ -246,7 +246,7 @@ Linux 镜像需要在 Linux 主机上收集和推送，但是你必须先将 Win ```plain helm repo add jetstack https://charts.jetstack.io helm repo update - helm fetch jetstack/cert-manager --version v1.11.0 + helm fetch jetstack/cert-manager helm template ./cert-manager-.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/other-installation-methods.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/other-installation-methods.md new file mode 100644 index 00000000000..24d8ac7d47c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/other-installation-methods.md @@ -0,0 +1,19 @@ +--- +title: 其他安装方式 +--- + +### 离线安装 + +按照[以下步骤](air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)在离线环境中安装 Rancher Server。 + +离线环境可以是 Rancher Server 离线安装、防火墙后面或代理后面。 + +### Docker 安装 + +[单节点 Docker 安装](rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md)适用于想要测试 Rancher 的用户。你无需使用 Helm 在 Kubernetes 集群上运行 Rancher，你可以使用 `docker run` 命令，把 Rancher Server 组件安装到单个节点上。 + +Docker 安装仅用于开发和测试环境。 + +由于只有一个节点和一个 Docker 容器，因此，如果该节点发生故障，由于其他节点上没有可用的 etcd 数据副本，你将丢失 Rancher Server 的所有数据。 + +Rancher backup operator 可将 Rancher 从单个 Docker 容器迁移到高可用 Kubernetes 集群上。详情请参见[把 Rancher 迁移到新集群](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher.md index da7bd9fb728..7ac7caca50c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher.md @@ -6,6 +6,8 @@ title: 3. 安装 Rancher ### 安装 Helm CLI + + 在具有 kubeconfig 的主机上安装 [Helm](https://helm.sh/docs/intro/install/) CLI 以访问 Kubernetes 集群： ``` @@ -31,7 +33,7 @@ kubectl create namespace cert-manager 安装 cert-manager 的 CustomResourceDefinitions： ``` -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download//cert-manager.crds.yaml ``` 使用 Helm 安装 cert-manager。请注意，cert-manager 还需要你配置代理，以防它需要与 Let's Encrypt 或其他外部证书颁发商进行通信： @@ -44,7 +46,7 @@ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/ ``` helm upgrade --install cert-manager jetstack/cert-manager \ - --namespace cert-manager --version v1.11.0 \ + --namespace cert-manager \ --set http_proxy=http://${proxy_host} \ --set https_proxy=http://${proxy_host} \ --set no_proxy=127.0.0.0/8\\,10.0.0.0/8\\,cattle-system.svc\\,172.16.0.0/12\\,192.168.0.0/16\\,.svc\\,.cluster.local diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md new file mode 100644 index 00000000000..1701dba329e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md @@ -0,0 +1,13 @@ +--- +title: 使用 HTTP 代理安装 Rancher +--- + +很多企业本地运行的服务器或虚拟机不能直接访问互联网，但是出于安全考虑，他们必须通过 HTTP(S) 代理连接到外部服务。本教程将分步介绍如何在这样的环境中进行高可用的 Rancher 安装。 + +另外，用户也可以在没有任何互联网访问的情况下离线设置 Rancher。详情请参见 [Rancher 官方文档](../air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)。 + +## 安装概要 + +1. [配置基础设施](set-up-infrastructure.md) +2. [配置 Kubernetes 集群](install-kubernetes.md) +3. [安装 Rancher](install-rancher.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure.md index 58414b8ed1d..f60e31b2c3e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure.md @@ -22,7 +22,7 @@ title: '1. 配置基础设施' 这些主机将通过 HTTP 代理连接到互联网。 -请确保你的节点满足[操作系统，容器运行时，硬件和网络](../../../../pages-for-subheaders/installation-requirements.md)的常规要求。 +请确保你的节点满足[操作系统，容器运行时，硬件和网络](../../installation-requirements/installation-requirements.md)的常规要求。 如需获取配置 Linux 节点的示例，请参见[在 Amazon EC2 中配置节点](../../../../how-to-guides/new-user-guides/infrastructure-setup/nodes-in-amazon-ec2.md)的教程。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md index 6e1ac8ee365..34ee707431b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md @@ -2,6 +2,8 @@ title: 证书故障排除 --- + + ### 如何确定我的证书格式是否为 PEM？ 你可以通过以下特征识别 PEM 格式： diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md new file mode 100644 index 00000000000..c2a63b86a5e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md @@ -0,0 +1,209 @@ +--- +title: 使用 Docker 将 Rancher 安装到单个节点中 +description: 在开发和测试环境中，你可以使用 Docker 安装。在单个 Linux 主机上安装 Docker，然后使用一个 Docker 容器部署 Rancher。 +--- + + + +Rancher 可以通过运行单个 Docker 容器进行安装。 + +在这种安装方案中，你需要将 Docker 安装到单个 Linux 主机，然后使用单个 Docker 容器将 Rancher 部署到主机中。 + +:::note 想要使用外部负载均衡器？ + +请参阅[使用外部负载均衡器的 Docker 安装](../../../../how-to-guides/advanced-user-guides/configure-layer-7-nginx-load-balancer.md)。 + +::: + +Rancher 的 Docker 安装仅推荐用于开发和测试环境中。Rancher 版本决定了能否将 Rancher 迁移到高可用集群。 + +Rancher backup operator 可将 Rancher 从单个 Docker 容器迁移到高可用 Kubernetes 集群上。详情请参见[把 Rancher 迁移到新集群](../../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)。 + +## Rancher 特权访问 + +当 Rancher Server 部署在 Docker 容器中时，容器内会安装一个本地 Kubernetes 集群供 Rancher 使用。为 Rancher 的很多功能都是以 deployment 的方式运行的，而在容器内运行容器是需要特权模式的，因此你需要在安装 Rancher 时添加 `--privileged` 选项。 + +## 操作系统，Docker，硬件和网络要求 + +请确保你的节点满足常规的[安装要求](../../installation-requirements/installation-requirements.md)。 + +## 1. 配置 Linux 主机 + +按照[要求](../../installation-requirements/installation-requirements.md)配置一个 Linux 主机，用于运行 Rancher Server。 + +## 2. 选择一个 SSL 选项并安装 Rancher + +出于安全考虑，使用 Rancher 时请使用 SSL（Secure Sockets Layer）。SSL 保护所有 Rancher 网络通信（如登录和与集群交互）的安全。 + +:::tip 你是否需要： + +- 使用代理。参见 [HTTP 代理配置](../../../../reference-guides/single-node-rancher-in-docker/http-proxy-configuration.md)。 +- 配置自定义 CA 根证书以访问服务。参见[自定义 CA 根证书](../../../../reference-guides/single-node-rancher-in-docker/advanced-options.md#自定义-ca-证书)。 +- 完成离线安装。参见 [离线：Docker 安装](../air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)。 +- 记录所有 Rancher API 的事务。参加 [API 审计](../../../../reference-guides/single-node-rancher-in-docker/advanced-options.md#api-审计日志)。 + +::: + +选择以下的选项之一： + +- [选项 A：使用 Rancher 生成的默认自签名证书](#选项-a使用-rancher-生成的默认自签名证书) +- [选项 B：使用你自己的证书 - 自签名](#选项-b使用你自己的证书---自签名) +- [选项 C：使用你自己的证书 - 可信 CA 签名的证书](#选项-c使用你自己的证书---可信-ca-签名的证书) +- [选项 D：Let's Encrypt 证书](#选项-dlets-encrypt-证书) +- [选项 E：Localhost 隧道，不使用证书](#选项-elocalhost-隧道不使用证书) + +### 选项 A：使用 Rancher 生成的默认自签名证书 + +如果你在不考虑身份验证的开发或测试环境中安装 Rancher，可以使用 Rancher 生成的自签名证书安装 Rancher。这种安装方式避免了自己生成证书的麻烦。 + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + --privileged \ + rancher/rancher:latest +``` + +### 选项 B：使用你自己的证书 - 自签名 +在你团队访问 Rancher Server 的开发或测试环境中，创建一个用于你的安装的自签名证书，以便团队验证他们对实例的连接。 + +:::note 先决条件： + +使用 [OpenSSL](https://www.openssl.org/) 或其他方法创建自签名证书。 + +- 证书文件的格式必须是 PEM。 +- 在你的证书文件中，包括链中的所有中间证书。你需要对你的证书进行排序，把你的证书放在最前面，后面跟着中间证书。如需查看示例，请参见[证书故障排除](../getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md)。 + +::: + +创建证书后，运行以下 Docker 命令以安装 Rancher。使用 `-v` 标志并提供证书的路径，以将证书挂载到容器中。 + +| 占位符 | 描述 | +| ------------------- | --------------------- | +| `` | 包含证书文件的目录的路径。 | +| `` | 完整证书链的路径。 | +| `` | 证书私钥的路径。 | +| `` | CA 证书的路径。 | + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + -v //:/etc/rancher/ssl/cert.pem \ + -v //:/etc/rancher/ssl/key.pem \ + -v //:/etc/rancher/ssl/cacerts.pem \ + --privileged \ + rancher/rancher:latest +``` + +### 选项 C：使用你自己的证书 - 可信 CA 签名的证书 + +在公开暴露应用的生产环境中，请使用由可信 CA 签名的证书，以避免用户收到证书安全警告。 + +不建议将 Docker 安装用于生产环境。这些说明仅适用于测试和开发。 + +:::note 先决条件： + +- 证书文件的格式必须是 PEM。 +- 在你的证书文件中，包括可信 CA 提供的所有中间证书。你需要对你的证书进行排序，把你的证书放在最前面，后面跟着中间证书。如需查看示例，请参见[证书故障排除](../getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md)。 + +::: + +获取证书后，运行以下 Docker 命令。 + +- 使用 `-v` 标志并提供证书的路径，以将证书挂载到容器中。因为你的证书是由可信的 CA 签名的，因此你不需要安装额外的 CA 证书文件。 +- 使用 `--no-cacerts` 作为容器的参数，以禁用 Rancher 生成的默认 CA 证书。 + +| 占位符 | 描述 | +| ------------------- | ----------------------------- | +| `` | 包含证书文件的目录的路径。 | +| `` | 完整证书链的路径。 | +| `` | 证书私钥的路径。 | + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + -v //:/etc/rancher/ssl/cert.pem \ + -v //:/etc/rancher/ssl/key.pem \ + --privileged \ + rancher/rancher:latest \ + --no-cacerts +``` + +### 选项 D：Let's Encrypt 证书 + +:::caution + +Let's Encrypt 对新证书请求有频率限制。因此，请限制创建或销毁容器的频率。详情请参见 [Let's Encrypt 官方文档 - 频率限制](https://letsencrypt.org/docs/rate-limits/)。 + +::: + +你也可以在生产环境中使用 [Let's Encrypt](https://letsencrypt.org/) 证书。Let's Encrypt 使用 HTTP-01 质询来验证你对域名的控制权。如果要确认你对该域名有控制权，你可将用于访问 Rancher 的主机名（例如 `rancher.mydomain.com`）指向运行的主机的 IP。你可通过在 DNS 中创建 A 记录，以将主机名绑定到 IP 地址。 + +不建议将 Docker 安装用于生产环境。这些说明仅适用于测试和开发。 + +:::note 先决条件： + +- Let's Encrypt 是联网服务。因此，在内网和离线环境中不能使用。 +- 在 DNS 中创建一条记录，将 Linux 主机 IP 地址绑定到要用于访问 Rancher 的主机名（例如，`rancher.mydomain.com`）。 +- 在 Linux 主机上打开 `TCP/80` 端口。Let's Encrypt 的 HTTP-01 质询可以来自任何源 IP 地址，因此端口 `TCP/80` 必须开放开所有 IP 地址。 + +::: + +满足先决条件后，你可以运行以下命令使用 Let's Encrypt 证书安装 Rancher。 + +| 占位符 | 描述 | +| ----------------- | ------------------- | +| `` | 你的域名地址 | + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + --privileged \ + rancher/rancher:latest \ + --acme-domain +``` + +### 选项 E：Localhost 隧道，不使用证书 + +如果你在开发或测试环境中安装 Rancher，且环境中有运行的 localhost 隧道解决方案（如 [ngrok](https://ngrok.com/)），不要生成证书。此安装选项不需要证书。 + +- 使用 `--no-cacerts` 作为参数，以禁用 Rancher 生成的默认 CA 证书。 + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + --privileged \ + rancher/rancher:latest \ + --no-cacerts +``` + +## 高级选项 + +使用 Docker 将 Rancher 安装到单个节点时，有如下几个可开启的高级选项： + +- 自定义 CA 证书 +- API 审计日志 +- TLS 设置 +- 离线环境 +- 持久化数据 +- 在同一个节点中运行 `rancher/rancher` 和 `rancher/rancher-agent` + +详情请参见[本页](../../../../reference-guides/single-node-rancher-in-docker/advanced-options.md)。 + +## 故障排除 + +如需了解常见问题及故障排除提示，请参见[本页](certificate-troubleshooting.md)。 + +## 后续操作 + +- **推荐**：检查单节点[备份](../../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-docker-installed-rancher.md)和[恢复](../../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-docker-installed-rancher.md)。你可能暂时没有需要备份的数据，但是我们建议你在常规使用 Rancher 后创建备份。 +- 创建 Kubernetes 集群：[配置 Kubernetes 集群](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/roll-back-docker-installed-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/roll-back-docker-installed-rancher.md index 32fbf2f4254..762ec2bd1a5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/roll-back-docker-installed-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/roll-back-docker-installed-rancher.md @@ -2,6 +2,8 @@ title: 回滚 Docker 安装的 Rancher --- + + 如果 Rancher 升级没有成功完成，你需要回滚到你在 [Docker 升级](upgrade-docker-installed-rancher.md)之前使用的 Rancher 设置。回滚可以恢复： - 先前版本的 Rancher。 @@ -78,7 +80,7 @@ docker pull rancher/rancher: --privileged \ rancher/rancher: ``` - 特权访问是[必须](../../../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#rancher-特权访问)的。 + 特权访问是[必须](rancher-on-a-single-node-with-docker.md#rancher-特权访问)的。 :::danger diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md index 68d23b34c33..7dbef6d2cab 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md @@ -13,7 +13,7 @@ title: 升级 Docker 安装的 Rancher ## 先决条件 - 在 Rancher 文档中**检查[已知升级问题](../../install-upgrade-on-a-kubernetes-cluster/upgrades.md#已知升级问题)**，了解升级 Rancher 时最需要注意的问题。你可以在 [GitHub](https://github.com/rancher/rancher/releases) 发布说明以及 [Rancher 论坛](https://forums.rancher.com/c/announcements/12)中找到每个 Rancher 版本的已知问题。不支持升级或升级到 [rancher-alpha 仓库](../../resources/choose-a-rancher-version.md#helm-chart-仓库)中的任何 Chart。 -- **[仅适用于离线安装](../../../../pages-for-subheaders/air-gapped-helm-cli-install.md)：为新的 Rancher Server 版本收集和推送镜像**。按照指南为你想要升级的目标 Rancher 版本[推送镜像到私有镜像仓库](../air-gapped-helm-cli-install/publish-images.md)。 +- **[仅适用于离线安装](../air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)：为新的 Rancher Server 版本收集和推送镜像**。按照指南为你想要升级的目标 Rancher 版本[推送镜像到私有镜像仓库](../air-gapped-helm-cli-install/publish-images.md)。 ## 占位符 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/add-tls-secrets.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/add-tls-secrets.md index dc58f216be0..87f320d0dca 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/add-tls-secrets.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/add-tls-secrets.md @@ -42,4 +42,4 @@ Rancher 启动时会检索配置的 `tls-ca` 密文。如果 Rancher 在运行 ## 更新私有 CA 证书 -按照[步骤](update-rancher-certificate.md)更新 [Rancher 高可用 Kubernetes 安装](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md)中的 Ingress，或从默认自签名证书切换到自定义证书。 \ No newline at end of file +按照[步骤](update-rancher-certificate.md)更新 [Rancher 高可用 Kubernetes 安装](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中的 Ingress，或从默认自签名证书切换到自定义证书。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md index b168ecdcb05..7ddfcef2d8e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md @@ -11,7 +11,7 @@ title: 选择 Rancher 版本 -如果 Rancher Server 是[安装在 Kubernetes 集群上](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md)的，Rancher Server 的安装，升级和回滚中，都是使用 Kubernetes 集群上的 Helm Chart 来安装 Rancher 的。因此，在准备安装或升级 Rancher 高可用时，必须添加包含用于安装 Rancher 的 Chart 的 Helm Chart 仓库。 +如果 Rancher Server 是[安装在 Kubernetes 集群上](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)的，Rancher Server 的安装，升级和回滚中，都是使用 Kubernetes 集群上的 Helm Chart 来安装 Rancher 的。因此，在准备安装或升级 Rancher 高可用时，必须添加包含用于安装 Rancher 的 Chart 的 Helm Chart 仓库。 请参见 [Helm 版本要求](helm-version-requirements.md)选择 Helm 版本来安装 Rancher。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/helm-version-requirements.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/helm-version-requirements.md index 273711a4de0..9a1fc4d1078 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/helm-version-requirements.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/helm-version-requirements.md @@ -4,7 +4,9 @@ title: Helm 版本要求 本文介绍 Helm 的要求。Helm 是用于把 Rancher 安装在高可用 Kubernetes 集群上的工具。 -> 我们已针对 Helm 3 更新了安装指南。如果你使用 Helm 2 进行安装，请参见 [Helm 2 迁移到 Helm 3 文档](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/)。[本文](/versioned_docs/version-2.0-2.4/pages-for-subheaders/helm2.md)提供了较早的使用 Helm 2 的 Rancher 高可用安装指南的副本。如果你如果无法升级到 Helm 3，可以使用这个说明安装。 +> 我们已针对 Helm 3 更新了安装指南。如果你使用 Helm 2 进行安装，请参见 [Helm 2 迁移到 Helm 3 文档](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/)。[本文](/versioned_docs/version-2.0-2.4/getting-started/installation-and-upgrade/advanced-options/advanced-use-cases/helm2/helm2.md)提供了较早的使用 Helm 2 的 Rancher 高可用安装指南的副本。如果你如果无法升级到 Helm 3，可以使用这个说明安装。 + + - 如需安装或升级 Rancher 2.5，请使用 Helm 3.2.x 或更高版本。 - Kubernetes 1.16 要求 Helm 2.16.0 或更高版本。如果使用的是默认 Kubernetes 版本，请参见[发行说明](https://github.com/rancher/rke/releases)获取所使用的 RKE 版本。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/local-system-charts.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/local-system-charts.md index 4048c5b2566..17c965ad095 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/local-system-charts.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/local-system-charts.md @@ -10,4 +10,4 @@ title: 离线安装中设置本地 System Charts `system-charts` 的一个本地副本已经打包到 `rancher/rancher` 容器中。为了在离线安装中使用这些功能，你需要使用额外的环境变量 `CATTLE_SYSTEM_CATALOG=bundled` 来运行 Rancher 安装命令，该环境变量告诉 Rancher 使用 Chart 的本地副本，而不是尝试从 GitHub 获取 Chart。 -带有 `system-charts` 的 Rancher 安装命令示例包含在 Docker 和 Helm 的[离线安装说明](../../../pages-for-subheaders/air-gapped-helm-cli-install.md)中。 +带有 `system-charts` 的 Rancher 安装命令示例包含在 Docker 和 Helm 的[离线安装说明](../other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)中。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/resources.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/resources.md new file mode 100644 index 00000000000..2862aba2cd2 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/resources.md @@ -0,0 +1,25 @@ +--- +title: 资源 +--- + +### Docker 安装 + +[单节点 Docker 安装](../other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md)适用于想要测试 Rancher 的用户。你无需使用 Helm 在 Kubernetes 集群上运行 Rancher，你可以使用 `docker run` 命令，把 Rancher Server 组件安装到单个节点上。 + +由于只有一个节点和一个 Docker 容器，因此，如果该节点发生故障，由于其他节点上没有可用的 etcd 数据副本，你将丢失 Rancher Server 的所有数据。 + +### 离线安装 + +按照[以下步骤](../other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)在离线环境中安装 Rancher Server。 + +离线环境可以是 Rancher Server 离线安装、防火墙后面或代理后面。 + +### 高级选项 + +安装 Rancher 时，有如下几个可开启的高级选项：每个安装指南中都提供了对应的选项。了解选项详情： + +- [自定义 CA 证书](custom-ca-root-certificates.md) +- [API 审计日志](../../../how-to-guides/advanced-user-guides/enable-api-audit-log.md) +- [TLS 设置](../installation-references/tls-settings.md) +- [etcd 配置](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md) +- [离线安装 Local System Chart](local-system-charts.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md index 3d11417cda6..4678487523e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md @@ -4,7 +4,7 @@ title: 更新 Rancher 证书 ## 更新私有 CA 证书 -按照以下步骤轮换[安装在 Kubernetes 集群上](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md)、由 Rancher 使用的 SSL 证书和私有 CA，或转用由私有 CA 签发的 SSL 证书。 +按照以下步骤轮换[安装在 Kubernetes 集群上](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)、由 Rancher 使用的 SSL 证书和私有 CA，或转用由私有 CA 签发的 SSL 证书。 步骤概述： diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/upgrade-cert-manager.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/upgrade-cert-manager.md index c2d8a5a59b9..3d825627ef3 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/upgrade-cert-manager.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/resources/upgrade-cert-manager.md @@ -2,6 +2,8 @@ title: 升级 Cert-Manager --- +Rancher 适配 API 版本 `cert-manager.io/v1` 并且在 cert-manager v1.13.1 版本上进行了测试。 + Rancher 使用 cert-manager 为 Rancher 高可用部署自动生成和续期 TLS 证书。从 2019 秋季开始，cert-manager 发生了以下的三个重要变更。如果你在此时间段前创建了 Rancher 高可用部署，请进行相关操作。 1. [从 2019 年 11 月 1 日开始，Let's Encrypt 已阻止低于 0.8.0 的 cert-manager 实例。](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753) @@ -92,8 +94,7 @@ Rancher 使用 cert-manager 为 Rancher 高可用部署自动生成和续期 TLS ```plain helm install \ cert-manager jetstack/cert-manager \ - --namespace cert-manager \ - --version v1.11.0 + --namespace cert-manager ``` 1. [恢复备份资源](https://cert-manager.io/docs/tutorials/backup/#restoring-resources)： @@ -125,7 +126,7 @@ Rancher 使用 cert-manager 为 Rancher 高可用部署自动生成和续期 TLS 1. 从 [Helm Chart 仓库](https://artifacthub.io/packages/helm/cert-manager/cert-manager)中获取最新可用的 cert-manager Chart： ```plain - helm fetch jetstack/cert-manager --version v1.11.0 + helm fetch jetstack/cert-manager ``` 1. 使用安装 Chart 的选项来渲染 cert-manager 模板。记住要设置 `image.repository` 选项，以从你的私有镜像仓库拉取镜像。此操作会创建一个包含 Kubernetes manifest 文件的 `cert-manager` 目录。 @@ -140,6 +141,8 @@ Rancher 使用 cert-manager 为 Rancher 高可用部署自动生成和续期 TLS --set cainjector.image.repository=/quay.io/jetstack/cert-manager-cainjector ``` + + Helm 2 命令如下： ```plain diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md index aa851f3a50c..b23f7651dd3 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md @@ -32,9 +32,9 @@ RKE v1.1.0 改变了集群升级的方式。 :::note 先决条件： -- 以下选项适用于 [Rancher 启动的 Kubernetes 集群](../../pages-for-subheaders/launch-kubernetes-with-rancher.md)和[注册的 K3s Kubernetes 集群](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/register-existing-clusters.md#已注册-rke2-和-k3s-集群的附加功能)。 +- 以下选项适用于 [Rancher 启动的 Kubernetes 集群](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)和[注册的 K3s Kubernetes 集群](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/register-existing-clusters.md#已注册-rke2-和-k3s-集群的附加功能)。 - 以下选项也适用于导入且已注册的 RKE2 集群。如果你从外部云平台导入集群但不注册，你将无法在 Rancher 中升级 Kubernetes 版本。 -- 在升级 Kubernetes 之前，先[备份你的集群](../../pages-for-subheaders/backup-restore-and-disaster-recovery.md)。 +- 在升级 Kubernetes 之前，先[备份你的集群](../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md)。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md index 2d9c0f00918..3d01c38aed8 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md @@ -10,7 +10,7 @@ Kubernetes API 可以在次要版本之间更改。因此，我们不支持引 ::: -Rancher 的 Kubernetes 元数据包含 Rancher 用于配置 [RKE 集群](../../pages-for-subheaders/launch-kubernetes-with-rancher.md)的 Kubernetes 版本信息。Rancher 会定期同步数据并为 **系统镜像**、**服务选项**和**插件模板**创建自定义资源定义 (CRD)。因此，当新的 Kubernetes 版本与 Rancher Server 版本兼容时，Kubernetes 元数据可以使 Rancher 使用新版本来配置集群。元数据概述了 [Rancher Kubernetes Engine](https://rancher.com/docs/rke/latest/en/) (RKE) 用于部署各种 Kubernetes 版本的信息。 +Rancher 的 Kubernetes 元数据包含 Rancher 用于配置 [RKE 集群](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)的 Kubernetes 版本信息。Rancher 会定期同步数据并为 **系统镜像**、**服务选项**和**插件模板**创建自定义资源定义 (CRD)。因此，当新的 Kubernetes 版本与 Rancher Server 版本兼容时，Kubernetes 元数据可以使 Rancher 使用新版本来配置集群。元数据概述了 [Rancher Kubernetes Engine](https://rancher.com/docs/rke/latest/en/) (RKE) 用于部署各种 Kubernetes 版本的信息。 下表描述了受周期性数据同步影响的 CRD。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/overview.md index 903c8c28bd2..be0cf58660c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/overview.md @@ -30,21 +30,21 @@ Rancher API Server 是基于嵌入式 Kubernetes API Server 和 etcd 数据库 ### 授权和基于角色的权限控制（RBAC） -- **用户管理**：Rancher API Server 除了管理本地用户，还[管理用户用来访问外部服务所需的认证信息](../pages-for-subheaders/authentication-config.md)，如登录 Active Directory 和 GitHub 所需的账号密码。 -- **授权**：Rancher API Server 可以管理[访问控制策略](../pages-for-subheaders/manage-role-based-access-control-rbac.md)和[安全策略](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md)。 +- **用户管理**：Rancher API Server 除了管理本地用户，还[管理用户用来访问外部服务所需的认证信息](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md)，如登录 Active Directory 和 GitHub 所需的账号密码。 +- **授权**：Rancher API Server 可以管理[访问控制策略](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md)和[安全策略](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md)。 ### 使用 Kubernetes 的功能 -- **配置 Kubernetes 集群**：Rancher API Server 可以在已有节点上[配置 Kubernetes](../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)，或进行 [Kubernetes 版本升级](installation-and-upgrade/upgrade-and-roll-back-kubernetes.md)。 -- **管理应用商店**：Rancher 支持使用 [Helm Chart 应用商店](../pages-for-subheaders/helm-charts-in-rancher.md)实现轻松重复部署应用。 -- **管理项目**：项目由集群中多个命名空间和访问控制策略组成，是 Rancher 中的一个概念，Kubernetes 中并没有这个概念。你可以使用项目实现以组为单位，管理多个命名空间，并进行 Kubernetes 相关操作。Rancher UI 提供用于[项目管理](../pages-for-subheaders/manage-projects.md)和[项目内应用管理](../pages-for-subheaders/kubernetes-resources-setup.md)的功能。 -- **Fleet 持续交付**：在 Rancher 中，你可以使用 [Fleet 持续交付](../pages-for-subheaders/fleet-gitops-at-scale.md)将应用程序从 Git 仓库部署到目标下游 Kubernetes 集群，无需任何手动操作。 -- **Istio**：[Rancher 与 Istio 集成](../pages-for-subheaders/istio.md)，使得管理员或集群所有者可以将 Istio 交给开发者，然后开发者使用 Istio 执行安全策略，排查问题，或为蓝绿部署，金丝雀部署，和 A/B 测试进行流量管理。 +- **配置 Kubernetes 集群**：Rancher API Server 可以在已有节点上[配置 Kubernetes](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)，或进行 [Kubernetes 版本升级](installation-and-upgrade/upgrade-and-roll-back-kubernetes.md)。 +- **管理应用商店**：Rancher 支持使用 [Helm Chart 应用商店](../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md)实现轻松重复部署应用。 +- **管理项目**：项目由集群中多个命名空间和访问控制策略组成，是 Rancher 中的一个概念，Kubernetes 中并没有这个概念。你可以使用项目实现以组为单位，管理多个命名空间，并进行 Kubernetes 相关操作。Rancher UI 提供用于[项目管理](../how-to-guides/advanced-user-guides/manage-projects/manage-projects.md)和[项目内应用管理](../how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md)的功能。 +- **Fleet 持续交付**：在 Rancher 中，你可以使用 [Fleet 持续交付](../integrations-in-rancher/fleet/fleet.md)将应用程序从 Git 仓库部署到目标下游 Kubernetes 集群，无需任何手动操作。 +- **Istio**：[Rancher 与 Istio 集成](../integrations-in-rancher/istio/istio.md)，使得管理员或集群所有者可以将 Istio 交给开发者，然后开发者使用 Istio 执行安全策略，排查问题，或为蓝绿部署，金丝雀部署，和 A/B 测试进行流量管理。 ### 配置云基础设施 - **同步节点信息**：Rancher API Server 可以同步所有集群中全部[节点](../how-to-guides/new-user-guides/manage-clusters/nodes-and-node-pools.md)的信息。 -- **配置云基础设施**：如果你为 Rancher 配置了云提供商，Rancher 可以在云端动态配置[新节点](../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md)和[持久化存储](../pages-for-subheaders/create-kubernetes-persistent-storage.md)。 +- **配置云基础设施**：如果你为 Rancher 配置了云提供商，Rancher 可以在云端动态配置[新节点](../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)和[持久化存储](../how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md)。 ### 查看集群信息 @@ -54,9 +54,9 @@ Rancher API Server 是基于嵌入式 Kubernetes API Server 和 etcd 数据库 ## 使用 Rancher 编辑下游集群 -对于已有集群而言，可提供的选项和设置取决于你配置集群的方法。例如，只有[通过 RKE 启动](../pages-for-subheaders/launch-kubernetes-with-rancher.md)的集群才有可编辑的**集群选项**。 +对于已有集群而言，可提供的选项和设置取决于你配置集群的方法。例如，只有[通过 RKE 启动](../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)的集群才有可编辑的**集群选项**。 -使用 Rancher 创建集群后，集群管理员可以管理集群成员，管理节点池，或进行[其他操作](../pages-for-subheaders/cluster-configuration.md)。 +使用 Rancher 创建集群后，集群管理员可以管理集群成员，管理节点池，或进行[其他操作](../reference-guides/cluster-configuration/cluster-configuration.md)。 下表总结了每一种类型的集群和对应的可编辑的选项和设置： diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/aws-marketplace.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/aws-marketplace.md index 2fc1ce0bd33..1d06135d7f4 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/aws-marketplace.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/aws-marketplace.md @@ -1,6 +1,6 @@ --- -title: Rancher AWS Marketplace 快速入门 -description: 使用 Amazon EKS 部署 Rancher Server。 +title: Rancher Prime AWS Marketplace 快速入门 +description: 使用 Amazon Marketplace 列表部署 Rancher Server。 --- -Amazon Elastic Kubernetes Service (EKS) 可以快速[将 Rancher 部署到 Amazon Web Services (AWS)](https://documentation.suse.com/trd/kubernetes/single-html/gs_rancher_aws-marketplace/)。详情请参见我们的 [Amazon Marketplace 列表](https://aws.amazon.com/marketplace/pp/prodview-go7ent7goo5ae)。观看 [demo](https://youtu.be/9dznJ7Ons0M)，了解 AWS Marketplace SUSE Rancher 设置的演练。 +你可以快速将 Rancher Prime 部署到 Amazon Elastic Kubernetes Service (EKS)。详情请参见[说明](https://suse-enceladus.github.io/marketplace-docs/rancher-prime/aws/?repository=rancher-payg-billing-adapter-llc-prd)以及我们的 [Amazon Marketplace 列表](https://aws.amazon.com/marketplace/pp/prodview-go7ent7goo5ae)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/aws.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/aws.md index cb47333ce32..34590ac8931 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/aws.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/aws.md @@ -7,7 +7,7 @@ description: 阅读此分步 Rancher AWS 指南，以快速部署带有单节点 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -86,7 +86,7 @@ AWS 模块只创建一个 EC2 密钥对、一个 EC2 安全组和一个 EC2 实 ## 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/azure.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/azure.md index 33a8d411ba0..2775340b7a5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/azure.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/azure.md @@ -7,7 +7,7 @@ description: 阅读此分步 Rancher Azure 指南，以快速部署带有单节 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -72,7 +72,7 @@ description: 阅读此分步 Rancher Azure 指南，以快速部署带有单节 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager.md new file mode 100644 index 00000000000..5eebb87e4e8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager.md @@ -0,0 +1,20 @@ +--- +title: 部署 Rancher Server +--- + +你可使用以下指南之一，在你选择的提供商中部署和配置 Rancher 和 Kubernetes 集群。 + +- [AWS](aws.md)（使用 Terraform） +- [AWS Marketplace](aws-marketplace.md)（使用 Amazon EKS） +- [Azure](azure.md)（使用 Terraform） +- [DigitalOcean](digitalocean.md)（使用 Terraform） +- [GCP](gcp.md)（使用 Terraform） +- [Hetzner Cloud](hetzner-cloud.md)（使用 Terraform） +- [Linode](linode.md) (使用 Terraform) +- [Vagrant](vagrant.md) +- [Equinix Metal](equinix-metal.md) +- [Outscale](outscale-qs.md)（使用 Terraform） + +如有需要，你可以查看以下指南以了解分步步骤。如果你需要在其他提供商中或本地运行 Rancher，或者你只是想看看它是多么容易上手，你可阅读以下指南： + +- [手动安装](helm-cli.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md index e87f17c26d0..8e0280f9fa6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md @@ -7,7 +7,7 @@ description: 阅读此分步 Rancher DigitalOcean 指南，以快速部署带有 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -65,7 +65,7 @@ description: 阅读此分步 Rancher DigitalOcean 指南，以快速部署带有 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/equinix-metal.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/equinix-metal.md index c7316352cc2..bca4519d296 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/equinix-metal.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/equinix-metal.md @@ -11,7 +11,7 @@ title: Rancher Equinix Metal 快速入门 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。不建议将 Docker 安装用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。不建议将 Docker 安装用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -38,7 +38,7 @@ title: Rancher Equinix Metal 快速入门 - 如果使用 CLI 或 API 配置新的 Equinix Metal Server，你需要提供项目 ID、计划、metro 和操作系统。 - 当使用云主机的虚拟机时，你需要允许 80 和 443 端口的入站 TCP 通信。有关端口配置的信息，请参见你的云主机的文档。 - 如需了解所有端口要求，请参见 [Docker 安装](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)。 -- 根据我们的[要求](../../../pages-for-subheaders/installation-requirements.md)配置主机。 +- 根据我们的[要求](../../installation-and-upgrade/installation-requirements/installation-requirements.md)配置主机。 ::: ### 2. 安装 Rancher @@ -103,4 +103,4 @@ title: Rancher Equinix Metal 快速入门 #### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md index 5c113fc447c..74af3068f04 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md @@ -7,7 +7,7 @@ description: 阅读此分步 Rancher GCP 指南，以快速部署带有单节点 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -68,7 +68,7 @@ description: 阅读此分步 Rancher GCP 指南，以快速部署带有单节点 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md index b3acb4f9ecc..b4f0395ac2d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md @@ -10,7 +10,7 @@ title: Helm CLI 快速入门 你的 Linux 主机可以位于任何地方。例如，它可以是 Amazon EC2 实例、Digital Ocean Droplet 或 Azure 虚拟机。其他 Rancher 文档也经常称它们为“节点”。部署 Linux 主机的一种方法是设置一个 Amazon EC2 实例，如[本教程](../../../how-to-guides/new-user-guides/infrastructure-setup/nodes-in-amazon-ec2.md)中所示。 -完整的安装要求在[这里](../../../pages-for-subheaders/installation-requirements.md)。 +完整的安装要求在[这里](../../installation-and-upgrade/installation-requirements/installation-requirements.md)。 ## 在 Linux 上安装 K3s @@ -104,7 +104,7 @@ helm repo add rancher-latest https://releases.rancher.com/server-charts/latest kubectl create namespace cattle-system -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download//cert-manager.crds.yaml helm repo add jetstack https://charts.jetstack.io @@ -112,14 +112,12 @@ helm repo update helm install cert-manager jetstack/cert-manager \ --namespace cert-manager \ - --create-namespace \ - --version v1.11.0 + --create-namespace # Windows Powershell helm install cert-manager jetstack/cert-manager ` --namespace cert-manager ` - --create-namespace ` - --version v1.11.0 + --create-namespace ``` 安装 Rancher 的最终命令如下。该命令需要一个将流量转发到 Linux 主机的域名。为了简化本教程，你可以使用假域名。`.sslip.io` 是一个假域名的例子。 @@ -149,6 +147,6 @@ helm install rancher rancher-latest/rancher ` 为了简化说明，我们使用了一个假域名和自签名证书来进行安装。因此，你可能需要在 Web 浏览器中添加一个安全例外来查看 Rancher UI。请注意，对于生产安装，你需要具有负载均衡器、真实域名和真实证书的高可用性设置。 -这些说明还省略了完整的安装要求和其他安装选项。如果你对这些步骤有任何疑问，请参阅完整的 [Helm CLI 安装文档](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md)。 +这些说明还省略了完整的安装要求和其他安装选项。如果你对这些步骤有任何疑问，请参阅完整的 [Helm CLI 安装文档](../../installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)。 -要使用新的 Rancher Server 来启动新的 Kubernetes 集群，你可能需要在 Rancher 中设置云凭证。有关更多信息，请参阅[使用 Rancher 启动 Kubernetes 集群](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md)。 +要使用新的 Rancher Server 来启动新的 Kubernetes 集群，你可能需要在 Rancher 中设置云凭证。有关更多信息，请参阅[使用 Rancher 启动 Kubernetes 集群](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md index 585353441b2..16ebb833b2a 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md @@ -7,7 +7,7 @@ description: 阅读此分步 Rancher Hetzner Cloud 指南，以快速部署带 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -67,7 +67,7 @@ description: 阅读此分步 Rancher Hetzner Cloud 指南，以快速部署带 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/linode.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/linode.md new file mode 100644 index 00000000000..9b6a00f6f7e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/linode.md @@ -0,0 +1,79 @@ +--- +title: Rancher Linode 快速入门指南 +description: 阅读此分步 Rancher Linode 指南，以快速部署带有单节点下游 Kubernetes 集群的 Rancher Server。 +--- + +你可以参考以下步骤，在 Linode 的单节点 K3s Kubernetes 集群中快速部署 Rancher Server，并附加一个单节点下游 Kubernetes 集群。 + +:::caution + +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 + +::: + +## 先决条件 + +:::caution + +部署到 Linode 会产生费用。 + +::: + +- [Linode 账号](https://linode.com): 用于运行服务器和集群。 +- [Linode 访问密钥](https://www.linode.com/docs/products/tools/api/guides/manage-api-tokens/): 用于权限认证的 Linode 访问密钥。 +- [Terraform](https://www.terraform.io/downloads.html): 用于在 Linode 中配置服务器和集群。 + + +## 开始使用 + +1. 使用命令行工具，执行 `git clone https://github.com/rancher/quickstart` 把 [Rancher Quickstart](https://github.com/rancher/quickstart) 克隆到本地。 + +2. 执行 `cd quickstart/rancher/linode` 命令，进入包含 Terraform 文件的 Linode 文件夹。 + +3. 把 `terraform.tfvars.example` 文件重命名为 `terraform.tfvars`。 + +4. 编辑 `terraform.tfvars` 文件，并替换以下变量： + - `linode_token` - 上面提到的 Linode 访问密钥。 + - `rancher_server_admin_password` - 替换为创建 Rancher Server 的 admin 账号的密码（最少 12 字符） + +5. **可选**：修改 `terraform.tfvars` 中的可选参数。 + 参见 [Quickstart Readme](https://github.com/rancher/quickstart) 以及 [Linode Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/linode) 了解更多信息。 + 建议包括： + - `linode_region` - 创建服务器以及集群的目标 Linode 区域。 + - 默认: `eu-central` + - 完整的区域列表, 请参照[官方的可用区域页面](https://www.linode.com/global-infrastructure/availability/). + - `prefix` - 所有创建资源的前缀 + - `linode_type` - 所有的 Linode 资源使用的类型/计划 + - 默认: `g6-standard-2` + - 完整的计划列表, 请参照[官方的计划类型页面](https://www.linode.com/docs/products/compute/compute-instances/plans/). + +6. 执行 `terraform init`。 + +7. 执行 `terraform apply --auto-approve` 以初始化环境。然后，等待命令行工具返回以下信息： + + ``` + Apply complete! Resources: 15 added, 0 changed, 0 destroyed. + + Outputs: + + rancher_node_ip = xx.xx.xx.xx + rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io + workload_node_ip = yy.yy.yy.yy + ``` + +8. 将以上输出中的 `rancher_server_url` 粘贴到浏览器中。在登录页面中登录（默认用户名为 `admin`，密码为在 `rancher_server_admin_password` 中设置的密码）。 +9. 使用 `quickstart/rancher/linode` 中生成的 `id_rsa` 密钥 SSH 到 Rancher Server。 + +#### 结果 + +两个 Kubernetes 集群已部署到你的 Linode 账户中，一个运行 Rancher Server，另一个为实验部署做好准备。请注意，虽然这种设置是探索 Rancher 功能的好方法，但在生产环境中，应遵循我们的高可用设置指南。用于虚拟机的 SSH 密钥是自动生成的，存储在模块目录中。 + +### 后续操作 + +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 + +## 销毁环境 + +1. 进入 `quickstart/rancher/linode` 文件夹，然后执行 `terraform destroy --auto-approve`。 + +2. 等待命令行界面显示资源已删除的消息。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md index 46b42901db9..76d320a862e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md @@ -7,7 +7,7 @@ description: 阅读此分步 Rancher Outscale 指南，以快速部署带有单 :::note -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -67,7 +67,7 @@ description: 阅读此分步 Rancher Outscale 指南，以快速部署带有单 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/vagrant.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/vagrant.md index dcddb74cfe9..1622c44a7b0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/vagrant.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/vagrant.md @@ -6,7 +6,7 @@ title: Vagrant 快速入门 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -42,7 +42,7 @@ title: Vagrant 快速入门 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md new file mode 100644 index 00000000000..80c8505d35e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md @@ -0,0 +1,8 @@ +--- +title: 部署工作负载 +--- + +这些指南指导你完成一个应用的部署，包括如何将应用暴露在集群之外使用。 + +- [部署带有 Ingress 的工作负载](workload-ingress.md) +- [部署带有 NodePort 的工作负载](nodeports.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/quick-start-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/quick-start-guides.md new file mode 100644 index 00000000000..e3e750dc9b4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/quick-start-guides/quick-start-guides.md @@ -0,0 +1,17 @@ +--- +title: Rancher 部署快速入门指南 +--- + +:::caution + +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../installation-and-upgrade/installation-and-upgrade.md)。 + +::: + +你可以阅读本章节，以快速开始部署和测试 Rancher 2.x。本章节包含 Rancher 的简单设置和一些常见用例的说明。未来，我们会在本章节中添加更多内容。 + +我们提供以下快速入门指南： + +- [部署 Rancher Server](deploy-rancher-manager/deploy-rancher-manager.md)：使用最方便的方式运行 Rancher。 + +- [部署工作负载](deploy-workloads/deploy-workloads.md)：部署一个简单的[工作负载](https://kubernetes.io/docs/concepts/workloads/)并公暴露工作负载，以从集群外部访问工作负载。 From 209f133f0423dced0c4c10d2b5d348e0abc0ae1a Mon Sep 17 00:00:00 2001 From: STARRY-S Date: Thu, 11 Apr 2024 04:51:02 +0800 Subject: [PATCH 42/67] Update missing Chinese translations of v2.8 and latest versions. (#1212) * Update missing latest Chinese translations Update latest version missing Chinese translations in `reference-guides` and `enable-api-audit-log-in-downstream-clusters.md` in `how-to-guides`. * Update missing v2.8 Chinese translations Update v2.8 version missing Chinese translations in `reference-guides` and `enable-api-audit-log-in-downstream-clusters.md` in `how-to-guides`. --- ...le-api-audit-log-in-downstream-clusters.md | 247 +++++++++++++ .../backup-restore-configuration.md | 12 + .../best-practices/best-practices.md | 21 ++ .../rancher-managed-clusters.md | 23 ++ .../rancher-server/rancher-server.md | 21 ++ ...and-best-practices-for-rancher-at-scale.md | 117 +++++++ .../cluster-configuration.md | 32 ++ .../downstream-cluster-configuration.md | 9 + .../machine-configuration.md | 9 + .../node-template-configuration.md | 9 + .../gke-cluster-configuration.md | 324 ++++++++++++++++++ .../rancher-server-configuration.md | 16 + .../use-existing-nodes/use-existing-nodes.md | 141 ++++++++ .../rancher-manager-architecture.md | 20 ++ .../single-node-rancher-in-docker.md | 9 + ...le-api-audit-log-in-downstream-clusters.md | 140 ++++++++ .../backup-restore-configuration.md | 12 + .../best-practices/best-practices.md | 21 ++ .../rancher-managed-clusters.md | 23 ++ .../rancher-server/rancher-server.md | 21 ++ ...and-best-practices-for-rancher-at-scale.md | 117 +++++++ .../cluster-configuration.md | 32 ++ .../downstream-cluster-configuration.md | 9 + .../machine-configuration.md | 9 + .../node-template-configuration.md | 9 + .../gke-cluster-configuration.md | 324 ++++++++++++++++++ .../rancher-server-configuration.md | 16 + .../use-existing-nodes/use-existing-nodes.md | 141 ++++++++ .../rancher-manager-architecture.md | 21 ++ .../single-node-rancher-in-docker.md | 9 + 30 files changed, 1914 insertions(+) create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/backup-restore-configuration/backup-restore-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/best-practices.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/rancher-server/rancher-server.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/cluster-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/downstream-cluster-configuration/downstream-cluster-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/machine-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/node-template-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/rancher-server-configuration/rancher-server-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/single-node-rancher-in-docker/single-node-rancher-in-docker.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/backup-restore-configuration/backup-restore-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/best-practices.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/rancher-server/rancher-server.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/cluster-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/downstream-cluster-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/machine-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/node-template-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/rancher-server-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/single-node-rancher-in-docker/single-node-rancher-in-docker.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters.md new file mode 100644 index 00000000000..747f302b79c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters.md @@ -0,0 +1,247 @@ +--- +title: 下游集群开启 API 审计日志 +--- + + + + + +Kubernetes 审计提供了由 Kube-apiserver 执行的与安全相关的、按时间顺序排列的集群审计记录。Kube API 会在请求执行的每个阶段都生成一个事件，然后根据策略进行预处理并保存，审计策略配置了要记录的内容。 + +你可能希望将审计日志配置为遵守互联网安全中心 (CIS) Kubernetes 基准控制的一部分。 + +有关配置的详细信息，请参阅 [Kubernetes 官方文档](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/)。 + + + + +### 方法1（建议）：设置 `machineGlobalConfig` 字段的 `audit-policy-file` + +你可以在配置文件中设置 `audit-policy-file`，Rancher 会将该文件保存在 Control Plane 节点的 `/var/lib/rancher/rke2/etc/config-files/audit-policy-file` 目录中，并在 RKE2 服务器中配置相应的选项。 + +例子： +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +spec: + rkeConfig: + machineGlobalConfig: + audit-policy-file: | + apiVersion: audit.k8s.io/v1 + kind: Policy + rules: + - level: RequestResponse + resources: + - group: "" + resources: + - pods +``` + +### 方法2：直接使用 `machineSelectorFiles` 和 `machineGlobalConfig` 配置 + +:::note + +Rancher v2.7.2 及以上版本提供此功能。 + +::: + +你可以使用 `machineSelectorFiles` 将审计策略文件传递到 Control Plane 节点，并使用 `machineGlobalConfig` 设置 kube-apiserver 的选项。 + +在此之前，你需要创建 [Secret](../new-user-guides/kubernetes-resources-setup/secrets.md) 或 [ConfigMap](../new-user-guides/kubernetes-resources-setup/configmaps.md) 作为审计策略的来源。 + +Secret 或 ConfigMap 必须满足以下要求： + +1. 必须位于 Cluster 对象所在的 `fleet-default` 命名空间中。 +2. 它必须含有 Annotation `rke.cattle.io/object-authorized-for-clusters: ,`，以允许目标集群使用它。 + +:::tip + +Rancher Dashboard 提供了易用的表单页面用于创建 Secret 或 ConfigMap。 + +::: + +例子： + +```yaml +apiVersion: v1 +data: + audit-policy: >- + IyBMb2cgYWxsIHJlcXVlc3RzIGF0IHRoZSBNZXRhZGF0YSBsZXZlbC4KYXBpVmVyc2lvbjogYXVkaXQuazhzLmlvL3YxCmtpbmQ6IFBvbGljeQpydWxlczoKLSBsZXZlbDogTWV0YWRhdGE= +kind: Secret +metadata: + annotations: + rke.cattle.io/object-authorized-for-clusters: cluster1 + name: + namespace: fleet-default +``` + +可以通过编辑集群 YAML 的 `machineSelectorFiles` 和 `machineGlobalConfig` 字段来启用和配置审计日志。 + +例子： + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +spec: + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - audit-policy-file=/dev-audit-policy.yaml + - audit-log-path=/dev-audit.logs + machineSelectorFiles: + - fileSources: + - configMap: + name: '' + secret: + items: + - key: audit-policy + path: /dev-audit-policy.yaml + name: dev-audit-policy + machineLabelSelector: + matchLabels: + rke.cattle.io/control-plane-role: 'true' +``` + +:::tip + +你还可以使用指令 `machineSelectorConfig` 和适当的 `machineLabelSelectors` 来达到相同的效果。 + +::: + +有关集群配置的更多信息，请参阅 [RKE2 集群配置参考](../../reference-guides/cluster-configuration/rancher-server-configuration/rke2-cluster-configuration.md)页面。 + + + + + +:::note + +Rancher v2.7.2 及以上版本提供此功能。 + +::: + +你可以使用 `machineSelectorFiles` 将审计策略文件传递到 Control Plane 节点，并使用 `machineGlobalConfig` 设置 kube-apiserver 的选项。 + +在此之前，你需要创建 [Secret](../new-user-guides/kubernetes-resources-setup/secrets.md) 或 [ConfigMap](../new-user-guides/kubernetes-resources-setup/configmaps.md) 作为审计策略的来源。 + +Secret 或 ConfigMap 必须满足以下要求： + +1. 必须位于 Cluster 对象所在的 `fleet-default` 命名空间中。 +2. 它必须含有 Annotation `rke.cattle.io/object-authorized-for-clusters: ,`，以允许目标集群使用它。 + +:::tip + +Rancher Dashboard 提供了易于使用的表单页面用于创建 [Secret](../new-user-guides/kubernetes-resources-setup/secrets.md) 或 [ConfigMap](../new-user-guides/kubernetes-resources-setup/configmaps.md)。 + +::: + +例子： + +```yaml +apiVersion: v1 +data: + audit-policy: >- + IyBMb2cgYWxsIHJlcXVlc3RzIGF0IHRoZSBNZXRhZGF0YSBsZXZlbC4KYXBpVmVyc2lvbjogYXVkaXQuazhzLmlvL3YxCmtpbmQ6IFBvbGljeQpydWxlczoKLSBsZXZlbDogTWV0YWRhdGE= +kind: Secret +metadata: + annotations: + rke.cattle.io/object-authorized-for-clusters: cluster1 + name: + namespace: fleet-default +``` + +可以通过编辑集群 YAML 的 `machineSelectorFiles` 和 `machineGlobalConfig` 字段来启用和配置审计日志。 + +例子： + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +spec: + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - audit-policy-file=/dev-audit-policy.yaml + - audit-log-path=/dev-audit.logs + machineSelectorFiles: + - fileSources: + - configMap: + name: '' + secret: + items: + - key: audit-policy + path: /dev-audit-policy.yaml + name: dev-audit-policy + machineLabelSelector: + matchLabels: + rke.cattle.io/control-plane-role: 'true' +``` + +:::tip + +你还可以使用指令 `machineSelectorConfig` 和适当的 `machineLabelSelectors` 来达到相同的效果。 + +::: + +有关集群配置的更多信息，请参阅 [K3s 集群配置参考](../../reference-guides/cluster-configuration/rancher-server-configuration/k3s-cluster-configuration.md) 页面。 + + + + + +可通过编辑集群 YAML 来启用和配置审计日志。 + +在启用审计日志后，将使用 RKE1 的默认值。 + +```yaml +# +# Rancher Config +# +rancher_kubernetes_engine_config: + services: + kube-api: + audit_log: + enabled: true +``` + +你还可以自定义审计日志配置。 + +```yaml +# +# Rancher Config +# +rancher_kubernetes_engine_config: + services: + kube-api: + audit_log: + enabled: true + configuration: + max_age: 6 + max_backup: 6 + max_size: 110 + path: /var/log/kube-audit/audit-log.json + format: json + policy: + apiVersion: audit.k8s.io/v1 # 这里必须填写 + kind: Policy + omitStages: + - "RequestReceived" + rules: + # Log pod changes at RequestResponse level + - level: RequestResponse + resources: + - group: "" + # Resource "pods" doesn't match requests to any subresource of pods, + # which is consistent with the RBAC policy. + resources: ["pods"] + # Log "pods/log", "pods/status" at Metadata level + - level: Metadata + resources: + - group: "" + resources: ["pods/log", "pods/status"] +``` + +配置详情请参考 [RKE1 官方文档](https://rke.docs.rancher.com/config-options/audit-log)。 + + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/backup-restore-configuration/backup-restore-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/backup-restore-configuration/backup-restore-configuration.md new file mode 100644 index 00000000000..6fbc1f202af --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/backup-restore-configuration/backup-restore-configuration.md @@ -0,0 +1,12 @@ +--- +title: Rancher 备份配置参考 +--- + + + + + +- [备份配置](backup-configuration.md) +- [还原配置](restore-configuration.md) +- [存储位置配置](storage-configuration.md) +- [Backup 和 Restore 自定义资源示例](examples.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/best-practices.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/best-practices.md new file mode 100644 index 00000000000..11f4ee82e41 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/best-practices.md @@ -0,0 +1,21 @@ +--- +title: 最佳实践 +--- + + + + + +本节介绍 Rancher 实现的最佳实践，其中包括对 Kubernetes、Docker、容器等技术的使用建议。最佳实践旨在利用 Rancher 及其客户的运营经验，助你更好地实现 Rancher。 + +如果你对用例的实际应用有任何疑问，请联系客户成功经理或支持中心。 + +你可以在左侧导航栏快速找到管理和部署 Rancher Server 的最佳实践。 + +如需查看更多最佳实践指南，请参见： + +- [安全类文档](../rancher-security/rancher-security.md) +- [Rancher 博客](https://www.suse.com/c/rancherblog/) +- [Rancher 论坛](https://forums.rancher.com/) +- [Rancher 用户的 Slack 群组](https://slack.rancher.io/) +- [B 站](https://space.bilibili.com/430496045/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md new file mode 100644 index 00000000000..e12f1db2498 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md @@ -0,0 +1,23 @@ +--- +title: Rancher 管理集群的最佳实践 +--- + + + + + +### Logging + +有关集群级别日志和应用日志的建议，请参见 [Logging 最佳实践](logging-best-practices.md)。 + +### Monitoring + +配置合理的监控和告警规则对于安全、可靠地运行生产环境中的工作负载至关重要。有关更多建议，请参阅[最佳实践](monitoring-best-practices.md)。 + +### 设置容器的技巧 + +配置良好的容器可以极大地提高环境的整体性能和安全性。有关容器设置的建议，请参见[设置容器的技巧](tips-to-set-up-containers.md)。 + +### Rancher 管理 vSphere 集群的最佳实践 + +[Rancher 管理 vSphere 集群的最佳实践](rancher-managed-clusters-in-vsphere.md)概述了在 vSphere 环境中配置下游 Rancher 集群的参考架构，以及 VMware 记录的标准 vSphere 最佳实践。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/rancher-server/rancher-server.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/rancher-server/rancher-server.md new file mode 100644 index 00000000000..d297b395a0a --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/rancher-server/rancher-server.md @@ -0,0 +1,21 @@ +--- +title: Rancher Server 的最佳实践 +--- + + + + + +本指南介绍了让 Rancher 管理下游 Kubernetes 集群的 Rancher Server 运行建议。 + +### 推荐的架构和基础设施 + +有关在高可用 Kubernetes 集群上设置 Rancher Server 的通用建议，请参见[本指南](tips-for-running-rancher.md)。 + +### 部署策略 + +[本指南](rancher-deployment-strategy.md)旨在帮助你选择部署策略（区域部署/中心辐射型部署），来让 Rancher Server 更好地管理下游 Kubernetes 集群。 + +### 在 vSphere 环境中安装 Rancher + +[本指南](on-premises-rancher-in-vsphere.md)介绍了在 vSphere 环境中安装 Rancher 的参考架构，以及 VMware 记录的标准 vSphere 最佳实践。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md new file mode 100644 index 00000000000..552e79ec76b --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md @@ -0,0 +1,117 @@ +--- +title: Rancher 大规模部署的调优和最佳实践 +--- + + + + + +本指南介绍了扩容 Rancher 相关场景的最佳实践和调优方法。随着系统规模的不断增长，性能自然会降低，但是可以采取一些步骤将 Rancher 的负载最小化，以优化 Rancher 管理较大基础设施的能力。 + +## 优化 Rancher 性能 + +* 及时升级 Rancher 的 Patch 更新。我们会不断的对 Rancher 进行性能增强和错误修复，最新的 Rancher 版本包含了基于开发人员和用户反馈的所有性能和稳定性的优化改进。 +* 逐渐的执行扩容操作，在操作的过程中观察并监控任何变化，在性能刚出现异常时发现问题并修复，避免其他干扰因素。 +* 尽可能地减少上游 Rancher 集群和下游集群之间的网络延时。需要注意的是，排除掉其他因素，延时与地理距离成正比，如果你的集群或节点分布在全球各地，请考虑改为安装多个 Rancher。 + +## 最小化上游集群的负载 + +在 Rancher 扩容时，一个典型的性能瓶颈是上游（local）Kubernetes 集群的资源增长。上游集群包含所有下游集群的数据信息，许多对下游集群的操作会在上游集群中创建新的对象，并需要在上游集群中进行计算处理。 + +### 最大限度地减少上游集群上的第三方软件 + +大规模运行 Rancher 可能会给内部 Kubernetes 组件（例如 `etcd` 或 `kubeapiserver`）带来大量负载。如果第三方软件干扰了这些组件或 Rancher 的性能，则可能会出现性能问题。 + +每个第三方软件都存在干扰性能的风险。为了防止上游集群出现性能问题，你应该避免运行除 Kubernetes 系统组件和 Rancher 本身之外的任何应用程序或组件。 + +以下类别的软件通常不会干扰 Rancher 或 Kubernetes 系统性能： + * Rancher 内部组件，例如 Fleet + * Rancher 扩展 + * 集群 API 组件 + * CNI 网络插件 + * 云控制器管理器 + * 观察和监控工具（`prometheus-rancher-exporter` 除外） + +除此之外，已发现以下软件会在 Rancher 扩容时影响性能： + * [CrossPlane](https://www.crossplane.io/) + * [Argo CD](https://argoproj.github.io/cd/) + * [Flux](https://fluxcd.io/) + * [prometheus-rancher-exporter](https://github.com/David-VTUK/prometheus-rancher-exporter)（请参阅 [issue 33](https://github.com/David-VTUK/prometheus-rancher-exporter/issues/33)） + +### 管理你的 Object 计数 + +Etcd 是 Kubernetes 和 Rancher 的后端数据库，该数据库可能会遇到单个 Kubernetes 资源类型数量的限制，确切的限制因素各不相同，取决于诸多因素。经验表明，一旦单个资源类型的对象数量超过 60000，通常会出现性能问题，通常该资源类型是 RoleBinding 对象。 + +这在 Rancher 中很常见，因为许多操作会在上游集群中创建新的 RoleBinding 对象。 + +你可以通过以下方法减少上游集群的 `RoleBinding` 数量： +* 减少使用 [Restricted Admin（受限管理员）](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md#受限管理员) 角色，改为使用其他角色。 +* 如果你使用了 [外部认证](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md)，配置其按组分配角色。 +* 仅在必要情况下将用户添加到集群和项目中。 +* 移除不需要的集群和项目。 +* 仅在必要情况下使用自定义角色。 +* 在自定义角色中，尽可能的减少规则数量。 +* 避免为用户添加多余的角色。 +* 考虑使用数量更少，但性能更强大的集群。 +* Kubernetes 权限始终是 “附加”（允许列表）而不是“减去”（拒绝列表）。尽量减少允许访问集群、项目或命名空间等方面的配置，因为这将导致创建大量的 RoleBinding 对象。 +* 实验一下，创建新项目或集群后，在你的特定用例场景是否表现出更少的 RoleBinding 的效果。 + +### RoleBinding 计数估计 + +预测给定的配置将创建多少个 RoleBinding 对象很复杂，但是以下注意事项可以提供粗略的估算： +* 对于最小的数量估计，请使用公式 `32C + U + 2UaC + 8P + 5Pa`。 + * `C` 是集群数量。 + * `U` 是用户数量。 + * `Ua` 是集群中具有成员资格的用户的平均数量。 + * `P` 是项目总数。 + * `Pa` 是项目中具有成员资格的平均用户数。 +* 受限管理员角色（Restricted Admin）遵循不同的公式，因为具有此角色的每个用户都会额外产生至少 `7C + 2P + 2` 个 `RoleBinding` 对象。 +* * `RoleBinding` 的数量会随着集群、项目和用户的数量线性增加。 + +### 使用新的应用代替旧版应用 + +Rancher 使用两个 Kubernetes 应用程序资源：`apps.projects.cattle.io` 和 `apps.cattle.cattle.io`。以`apps.projects.cattle.io` 为代表的旧版应用程序在低版本的 Cluster Manager UI 中引入，现已弃用。新的应用程序（由 `apps.catalog.cattle.io` 表示）可以在 Cluster Explorer UI 中安装部署。新的 `apps.cattle.cattle.io` 应用程序数据保存在下游集群中，这可以减少上游集群中的资源数量。 + +你应当删除 Cluster Manager UI 中遗留的所有旧版应用程序，并将其替换为 Cluster Explorer UI 中的应用程序。只在 Cluster Explorer UI 中创建任何新应用程序。 + +### 使用授权集群端点 (ACE) + +[授权集群端点](../../../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters.md#4-授权集群端点) (ACE) 提供了 Rancher 部署的 RKE、RKE2 和 K3s 集群的 Kubernetes API 访问。启用后，ACE 会为生成的 `kubeconfig` 文件配置直接访问下游集群 Endpoint，从而绕过 Rancher 代理。在可以直接访问下游集群 Kubernetes API 的场景下，可以减少 Rancher 负载。有关更多信息，请参阅[授权集群端点](../../../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters.md#4-授权集群端点)配置说明。 + +### 减少 Event Handler 执行 + +Rancher 的大部分逻辑发生在 Event Handler 上。每当资源对象产生更新或 Rancher 启动时，这些资源对应的 Event Handler 都会执行。除此之外，它们会每隔 15 小时在 Rancher 计划的同步缓存时再运行一次，这可能会导致 Rancher 运行过程中出现大量性能消耗。可使用 `CATTLE_SYNC_ONLY_CHANGED_OBJECTS` 环境变量禁用计划的 Handler 处理程序执行。如果每 15 小时出现一次性能资源高峰，此设置会有所帮助。 + +`CATTLE_SYNC_ONLY_CHANGED_OBJECTS` 的值可被设置为以下内容，以逗号分隔。这些值代表了处理程序的种类，将处理程序添加到该变量会禁止其在定期的缓存重新同步过程中运行。 + +* `mgmt`：在 Rancher 节点上运行的 Management 管理控制器。 +* `user`：所有集群运行的用户控制器。其中一部分与 Management 管理控制器运行在同一节点上，而另一部分运行在下游集群中，该选项针对的是前者。 +* `scaled`：每个 Rancher 节点上运行的规模控制器。因规模控制器负责关键功能，应当避免设置此值。如果设置可能会破坏集群稳定。 + +简而言之，如果你发现 CPU 使用率每 15 小时出现一次峰值，请将 `CATTLE_SYNC_ONLY_CHANGED_OBJECTS` 环境变量添加到 Rancher Deployment 中（添加至 `spec.containers.env` 列表），其值为 `mgmt,user`。 + +## Rancher 之外的优化 + +集群底层自身的配置也是影响性能的重要因素。如果上游集群存在错误配置，会带来 Rancher 软件所无法解决的性能瓶颈。 + +### 使用 RKE2 直接管理上游集群节点 + +由于 Rancher 对上游集群的要求非常高，尤其是在大规模部署场景，你需要拥有上游集群和节点的所有管理员权限，要找出资源消耗过高的根本原因，请使用标准的 Linux 故障排除工具，这有助于区分是 Rancher、Kubernetes 还是操作系统组件出现的问题。 + +尽管托管 Kubernetes 服务使部署和运行 Kubernetes 集群变得更加容易，但在大规模场景中，不鼓励将其用于上游集群。 托管 Kubernetes 服务通常会限制对单个节点和服务配置的访问。 + +建议在大规模用例场景中使用 RKE2 集群。 + +### 及时更新 Kubernetes 版本 + +你应当及时更新上游集群的 Kubernetes 版本，以确保你的集群具备最新的性能增强和问题修复。 + +### 优化 Etcd + +Etcd 是 Kubernetes 和 Rancher 的后端数据库，在 Rancher 性能中扮演重要的角色。 + +[Etcd 性能](https://etcd.io/docs/v3.4/op-guide/performance/)的两个主要瓶颈是磁盘和网络速度。Etcd 应当在具有高速网络和高读写速度 (IOPS) SSD 硬盘的专用节点上运行。有关 etcd 性能的更多信息，请参阅 [etcd 性能缓慢（性能测试和优化）](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装进行 etcd 调优](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md)。有关磁盘的信息可以在[安装要求](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#磁盘)中找到。 + +根据 etcd 的[复制机制](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size)，建议在三个节点上运行 etcd，运行在更多的节点上反而会降低速度。 + +Etcd 性能也会受节点之间的网络延迟影响，因此 etcd 节点应与 Rancher 节点部署在一起。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/cluster-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/cluster-configuration.md new file mode 100644 index 00000000000..111aa5afbf6 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/cluster-configuration.md @@ -0,0 +1,32 @@ +--- +title: 集群配置 +--- + + + + + +使用 Rancher 配置 Kubernetes 集群后，你仍然可以编辑集群的选项和设置。 + +有关编辑集群成员资格的信息，请转至[此页面](../how-to-guides/new-user-guides/manage-clusters/access-clusters/add-users-to-clusters.md)。 + +### 集群配置参考 + +集群配置选项取决于 Kubernetes 集群的类型： + +- [RKE 集群配置](rancher-server-configuration/rke1-cluster-configuration.md) +- [RKE2 集群配置](rancher-server-configuration/rke2-cluster-configuration.md) +- [K3s 集群配置](rancher-server-configuration/k3s-cluster-configuration.md) +- [EKS 集群配置](rancher-server-configuration/eks-cluster-configuration.md) +- [GKE 集群配置](rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md) +- [AKS 集群配置](rancher-server-configuration/aks-cluster-configuration.md) + +### 不同类型集群的管理功能 + +对于已有集群而言，可提供的选项和设置取决于你配置集群的方法。 + +下表总结了每一种类型的集群和对应的可编辑的选项和设置： + +import ClusterCapabilitiesTable from '../../shared-files/_cluster-capabilities-table.md'; + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/downstream-cluster-configuration/downstream-cluster-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/downstream-cluster-configuration/downstream-cluster-configuration.md new file mode 100644 index 00000000000..c3482cad893 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/downstream-cluster-configuration/downstream-cluster-configuration.md @@ -0,0 +1,9 @@ +--- +title: 下游集群配置 +--- + + + + + +以下文档将讨论[节点模板配置](node-template-configuration/node-template-configuration.md)和[主机配置](machine-configuration/machine-configuration.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/machine-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/machine-configuration.md new file mode 100644 index 00000000000..7669ec23377 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/machine-configuration.md @@ -0,0 +1,9 @@ +--- +title: 主机配置 +--- + + + + + +主机配置指的是如何将资源分配给虚拟机。请参阅 [Amazon EC2](amazon-ec2.md)、[DigitalOcean](digitalocean.md) 和 [Azure](azure.md) 的文档以了解更多信息。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/node-template-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/node-template-configuration.md new file mode 100644 index 00000000000..35779a5d3ae --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/node-template-configuration.md @@ -0,0 +1,9 @@ +--- +title: 节点模板配置 +--- + + + + + +要了解节点模板配置，请参阅[EC2 节点模板配置](amazon-ec2.md)、[DigitalOcean 节点模板配置](digitalocean.md)、[Azure 节点模板配置](azure.md)、[vSphere 节点模板配置](vsphere.md)和 [Nutanix 节点模板配置](nutanix.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md new file mode 100644 index 00000000000..5775e381b1c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md @@ -0,0 +1,324 @@ +--- +title: GKE 集群配置参考 +--- + + + + + +## Rancher 2.6 变更 + +- 支持额外的配置选项： + - 项目网络隔离 + - 网络标签 + +## 集群位置 + +| 值 | 描述 | +|--------|--------------| +| 位置类型 | 地区 (zone) 或区域 (region)。借助 GKE，你可以根据工作负载的可用性要求和预算创建一个量身定制的集群。默认情况下，集群的节点在单个计算区域中运行。选择多个区域时，集群的节点将跨越多个计算区域，而 controlplane 则只位于单个区域中。区域集群也增加了 controlplane 的可用性。有关选择集群可用性类型的帮助，请参阅[这些文档](https://cloud.google.com/kubernetes-engine/docs/best-practices/scalability#choosing_a_regional_or_zonal_control_plane)。 | +| 地区 | 计算引擎中的每个区域都包含多地区。有关可用区域和可用区的更多信息，请参阅[这些文档](https://cloud.google.com/compute/docs/regions-zones#available)。 | +| 其他地区 | 对于地区性集群，你可以选择其他地区来创建[多地区集群](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#multi-zonal_clusters)。 | +| 区域 | 对于[区域性集群](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#regional_clusters)，你可以选择一个区域。有关可用区域和可用区的更多信息，请参阅[本节](https://cloud.google.com/compute/docs/regions-zones#available)。地区名称的前面部分是区域的名称。 | + +## 集群选项 + +### Kubernetes 版本 + +_可变：是_ + +有关 GKE Kubernetes 版本的更多信息，请参阅[这些文档](https://cloud.google.com/kubernetes-engine/versioning)。 + +### 容器地址范围 + +_可变：否_ + +集群中 Pod 的 IP 地址范围。必须是有效的 CIDR 范围，例如 10.42.0.0/16。如果未指定，则会自动从 10.0.0.0/8 中选择一个随机范围，并排除已分配给 VM、其他集群或路由的范围。自动选择的范围可能与预留的 IP 地址、动态路由或与集群对等的 VPC 中的路由发生冲突。 + +### 网络 + +_可变：否_ + +集群连接的 Compute Engine 网络。将使用此网络创建路由和防火墙。如果使用[共享 VPC](https://cloud.google.com/vpc/docs/shared-vpc)，与你的项目共享的 VPC 网络将显示在此处。你将可以在此字段中进行选择。有关详细信息，请参阅[此页面](https://cloud.google.com/vpc/docs/vpc#vpc_networks_and_subnets)。 + +### 节点子网/子网 + +_可变：否_ + +集群连接到的 Compute Engine 子网。该子网必须属于**网络**字段中指定的网络。选择一个现有的子网，或选择“自动创建子网”来自动创建一个子网。如果不使用现有网络，则需要使用**子网名称**来生成一个。如果使用[共享 VPC](https://cloud.google.com/vpc/docs/shared-vpc)，与你的项目共享的 VPC 子网将显示在此处。如果使用共享 VPC 网络，则无法选择“自动创建子网”。如需更多信息，请参阅[此页面](https://cloud.google.com/vpc/docs/vpc#vpc_networks_and_subnets)。 + +### 子网名称 + +_可变：否_ + +使用提供的名称自动创建子网。如果为**节点子网**或**子网**选择了“自动创建子网”，则为必填。有关子网的更多信息，请参阅[此页面](https://cloud.google.com/vpc/docs/vpc#vpc_networks_and_subnets)。 + +### IP 别名 + +_可变：否_ + +启用[别名 IP](https://cloud.google.com/vpc/docs/alias-ip)。这将启用 VPC 原生流量路由。如果使用[共享 VPC](https://cloud.google.com/vpc/docs/shared-vpc)，则为必填。 + +### 网络策略 + +_可变：是_ + +在集群上启用的网络策略。网络策略定义了集群中 pod 和 service 之间可以发生的通信级别。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/how-to/network-policy)。 + +### 项目网络隔离 + +_可变：是_ + +选择启用或禁用项目间通信。请注意，如果启用**项目网络隔离**，则将自动启用**网络策略**和**网络策略配置**，反之则不然。 + +### 节点 IPv4 CIDR 块 + +_可变：否_ + +此集群中实例 IP 的 IP 地址范围。如果为**节点子网**或**子网**选择了“自动创建子网”，则可以进行设置。必须是有效的 CIDR 范围，例如 10.96.0.0/14。有关如何确定 IP 地址范围的详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing)。 + +### 集群次要范围名称 + +_可变：否_ + +Pod IP 地址的现有次要范围的名称。如果选中，将自动填充**集群 Pod 地址范围**。如果使用共享 VPC 网络，则为必填。 + +### 集群 Pod 地址范围 + +_可变：否_ + +分配给集群中 pod 的 IP 地址范围。必须是有效的 CIDR 范围，例如 10.96.0.0/11。如果未提供，将自动创建。如果使用共享 VPC 网络，则必须提供。有关如何确定 pod 的 IP 地址范围的更多信息，请参阅[本节](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing_secondary_range_pods)。 + +### Service 次要范围名称 + +_可变：否_ + +Service IP 地址的现有次要范围的名称。如果选中，将自动填充 **Service 地址范围**。如果使用共享 VPC 网络，则为必填。 + +### Service 地址范围 + +_可变：否_ + +分配给集群中 Service 的地址范围。必须是有效的 CIDR 范围，例如 10.94.0.0/18。如果未提供，将自动创建。如果使用共享 VPC 网络，则必须提供。有关如何确定 Service 的 IP 地址范围的详细信息，请参阅[本节](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing_secondary_range_svcs)。 + +### 私有集群 + +_可变：否_ + +:::caution + +私有集群需要在 Rancher 之外进行额外的规划和配置。请参阅[私有集群指南](gke-private-clusters.md)。 + +::: + +仅分配节点内部 IP 地址。除非在 GCP 中执行了额外的联网步骤，否则私有集群节点无法访问公共互联网。 + +### 启用私有端点 + +:::caution + +私有集群需要在 Rancher 之外进行额外的规划和配置。请参阅[私有集群指南](gke-private-clusters.md)。 + +::: + +_可变：否_ + +锁定对 controlplane 端点的外部访问。仅当**私有集群**也被选中时可用。如果选中，并且 Rancher 无法直接访问集群所在的虚拟私有云网络，Rancher 将提供在集群上运行的注册命令，以使 Rancher 能够连接到集群。 + +### 主 IPV4 CIDR 块 + +_可变：否_ + +controlplane VPC 的 IP 范围。 + +### 主授权网络 + +_可变：是_ + +启用 controlplane 授权网络，以阻止不受信任的非 GCP 源 IP 通过 HTTPS 访问 Kubernetes master。如果选择，则可以添加额外的授权网络。如果集群是使用公共端点创建的，则此选项可用于将公共端点的访问锁定到特定网络（例如运行 Rancher 服务的网络）。如果集群只有一个私有端点，则需要此设置。 + +## 其他选项 + +### 集群插件 + +其他 Kubernetes 集群组件。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters#Cluster.AddonsConfig)。 + +#### 水平 Pod 自动缩放 + +_可变：是_ + +Horizo​​ntal Pod Autoscaler 通过自动增加或减少 Pod 的数量来调整 Kubernetes 工作负载，从而响应工作负载的 CPU 或内存消耗，以及 Kubernetes 内部报告的自定义指标或集群外部设置的指标。详情请参见[本页面](https://cloud.google.com/kubernetes-engine/docs/concepts/horizontalpodautoscaler)。 + +#### HTTP (L7) 负载均衡 + +_可变：是_ + +HTTP (L7) 负载均衡将 HTTP 和 HTTPS 流量分配到托管在 GKE 上的后端。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer)。 + +#### 网络策略配置（仅限 master） + +_可变：是_ + +NetworkPolicy 的配置。仅跟踪 master 节点上是否启用了插件，不跟踪是否为节点启用了网络策略。 + +### 集群特征（Alpha 功能） + +_可变：否_ + +打开集群的所有 Kubernetes alpha API 组和功能。启用后，集群无法升级，并且会在 30 天后自动删除。由于 GKE SLA 未支持 alpha 集群，因此不建议将 Alpha 集群用于生产环境。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/concepts/alpha-clusters)。 + +### Logging 服务 + +_可变：是_ + +集群用于写入日志的日志管理服务。要么使用 [Cloud Logging](https://cloud.google.com/logging)，要么不使用日志管理服务（不会从集群中导出日志）。 + +### 监控服务 + +_可变：是_ + +集群用于写入指标的监控服务。要么使用 [Cloud Monitoring](https://cloud.google.com/monitoring)，要么不使用集群监控服务（不会从集群中导出指标）。 + + +### 维护窗口 + +_可变：是_ + +设置时长 4 小时的维护窗口的开始时间。使用 HH:MM 格式在 UTC 时区中指定时间。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/concepts/maintenance-windows-and-exclusions)。 + +## 节点池 + +在此部分中，输入描述节点池中每个节点的配置的详细信息。 + +### Kubernetes 版本 + +_可变：是_ + +节点池中每个节点的 Kubernetes 版本。有关 GKE Kubernetes 版本的更多信息，请参阅[这些文档](https://cloud.google.com/kubernetes-engine/versioning)。 + +### 镜像类型 + +_可变：是_ + +节点操作系统镜像。有关 GKE 为每个操作系统提供的节点镜像选项，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#available_node_images)。 + +:::note + +默认选项是 “Container-Optimized OS with Docker”。GCP Container-Optimized OS 上的只读文件系统与 Rancher 中的 [legacy logging](/versioned_docs/version-2.0-2.4/pages-for-subheaders/cluster-logging.md) 实现不兼容。如果你需要使用旧版日志管理功能，请选择 “Ubuntu with Docker” 或 “Ubuntu with Containerd”。[current logging feature](../../../../integrations-in-rancher/logging/logging.md) 与 Container-Optimized OS 镜像兼容。 + +::: + +:::note + +如果节点池镜像类型选择 “Windows Long Term Service Channel” 或 “Windows Semi-Annual Channel”，还必须至少添加一个 Container-Optimized OS 或 Ubuntu 节点池。 + +::: + +### 主机类型 + +_可变：否_ + +节点实例可用的虚拟化硬件资源。有关 Google Cloud 主机类型的详细信息，请参阅[此页面](https://cloud.google.com/compute/docs/machine-types#machine_types)。 + +### 根磁盘类型 + +_可变：否_ + +标准永久性磁盘由标准磁盘驱动器 (HDD) 支持，而 SSD 永久性磁盘由固态硬盘 (SSD) 支持。有关详细信息，请参阅[本节](https://cloud.google.com/compute/docs/disks)。 + +### 本地 SSD 磁盘 + +_可变：否_ + +配置每个节点的本地 SSD 磁盘存储（以 GB 为单位）。本地 SSD 物理连接到托管你的 VM 实例的服务器。与标准永久性磁盘或 SSD 永久性磁盘相比，本地 SSD 具有更高的吞吐量和更低的延迟。存储在本地 SSD 上的数据只会保留到实例停止或删除。有关详细信息，请参阅[本节](https://cloud.google.com/compute/docs/disks#localssds)。 + +### 抢占式节点（beta） + +_可变：否_ + +抢占式节点也称为抢占式虚拟机。通常是最长持续 24 小时的 Compute Engine 虚拟机实例，不提供可用性保证。详情请参见[本页面](https://cloud.google.com/kubernetes-engine/docs/how-to/preemptible-vms)。 + +### 污点 + +_可变：否_ + +将污点应用于节点时，仅允许容忍该污点的 Pod 在该节点上运行。在 GKE 集群中，你可以将污点应用到节点池，这会将污点应用到池中的所有节点。 + +### 节点标签 + +_可变：否_ + +你可以将标签应用到节点池，这会将标签应用到池中的所有节点。 + +无效标签会阻止升级，或阻止 Rancher 启动。有关标签语法的详细信息，请参阅 [Kubernetes 文档](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set)。 + +### 网络标签 + +_可变：否_ + +你可以将网络标签添加到节点池以制定防火墙规则和子网之间的路由。标签将应用于池中的所有节点。 + +有关标签语法和要求的详细信息，请参阅 [Kubernetes 文档](https://cloud.google.com/vpc/docs/add-remove-network-tags)。 + +## 组详细信息 + +在此部分中，输入描述节点池的详细信息。 + +### 名称 + +_可变：否_ + +输入节点池的名称。 + +### 初始节点数 + +_可变：是_ + +节点池中初始节点数的整数。 + +### 每个节点的最大 Pod 数量 + +_可变：否_ + +GKE 的硬性限制是每个节点 110 个 Pod。有关 Kubernetes 限制的更多信息，请参阅[本节](https://cloud.google.com/kubernetes-engine/docs/best-practices/scalability#dimension_limits)。 + +### 自动缩放 + +_可变：是_ + +节点池自动缩放会根据工作负载的需求动态创建或删除节点。详情请参见[本页面](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler)。 + +### 自动修复 + +_可变：是_ + +GKE 的节点自动修复功能可帮助你将集群中的节点保持在健康的运行状态。启用后，GKE 会定期检查集群中每个节点的运行状况。如果某个节点在较长时间段内连续未通过健康检查，GKE 会为该节点启动修复过程。有关详细信息，请参阅[自动修复节点](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair)。 + +### 自动升级 + +_可变：是_ + +启用后，当你的 controlplane [按照你的需求更新](https://cloud.google.com/kubernetes-engine/upgrades#automatic_cp_upgrades)时，自动升级功能会使集群中的节点与集群 controlplane（master）版本保持同步。有关自动升级节点的更多信息，参见[此页面。](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades) + +### 访问范围 + +_可变：否_ + +设置访问范围是为你的节点指定权限的旧版方法。 + +- **允许默认访问**：新集群的默认访问是 [Compute Engine 默认 ServiceAccount](https://cloud.google.com/compute/docs/access/service-accounts?hl=en_US#default_service_account)。 +- **允许完全访问所有 Cloud API**：通常，你只需设置云平台访问范围来允许完全访问所有 Cloud API，然后仅授予 ServiceAccount 相关的 IAM 角色。授予虚拟机实例的访问范围和授予 ServiceAccount 的 IAM 角色的组合决定了 ServiceAccount 对该实例的访问量。 +- **为每个 API 设置访问权限**：或者，你可以设置服务将调用的特定 API 方法的访问范围。 + +有关详细信息，请参阅[为 VM 启用 ServiceAccount](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances)。 + + +### 配置刷新间隔 + +刷新间隔可以通过 “gke-refresh” 来配置，它是一个代表秒的整数。 + +默认值为 300 秒。 + +你可以通过运行 `kubectl edit setting gke-refresh` 来更改同步间隔。 + +刷新窗口越短，争用条件发生的可能性就越小。但这确实增加了遇到 GCP API 可能存在的请求限制的可能性。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/rancher-server-configuration/rancher-server-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/rancher-server-configuration/rancher-server-configuration.md new file mode 100644 index 00000000000..4efbf1bb06b --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/rancher-server-configuration/rancher-server-configuration.md @@ -0,0 +1,16 @@ +--- +title: Rancher Server 配置 +--- + + + + + +- [RKE1 集群配置](rke1-cluster-configuration.md) +- [RKE2 集群配置](rke2-cluster-configuration.md) +- [K3s 集群配置](k3s-cluster-configuration.md) +- [EKS 集群配置](eks-cluster-configuration.md) +- [AKS 集群配置](aks-cluster-configuration.md) +- [GKE 集群配置](gke-cluster-configuration/gke-cluster-configuration.md) +- [使用现有节点](use-existing-nodes/use-existing-nodes.md) +- [同步集群](sync-clusters.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md new file mode 100644 index 00000000000..555b50a4a41 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md @@ -0,0 +1,141 @@ +--- +title: 在现有自定义节点上启动 Kubernetes +description: 要创建具有自定义节点的集群，你需要访问集群中的服务器，并根据 Rancher 的要求配置服务器。 +--- + + + + + +创建自定义集群时，Rancher 使用 RKE（Rancher Kubernetes Engine）在本地裸机服务器、本地虚拟机或云服务器节点中创建 Kubernetes 集群。 + +要使用此选项，你需要访问要在 Kubernetes 集群中使用的服务器。请根据[要求](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)配置每台服务器，其中包括硬件要求和 Docker 要求。在每台服务器上安装 Docker 后，你还需要在每台服务器上运行 Rancher UI 中提供的命令，从而将每台服务器转换为 Kubernetes 节点。 + +本节介绍如何设置自定义集群。 + +## 使用自定义节点创建集群 + +:::note 使用 Windows 主机作为 Kubernetes Worker 节点？ + +在开始之前，请参阅[配置 Windows 自定义集群](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)。 + +::: + +### 1. 配置 Linux 主机 + +你可以通过配置 Linux 主机，来创建自定义集群。你的主机可以是： + +- 云虚拟机 +- 本地虚拟机 +- 裸机服务器 + +如果要重复使用之前的自定义集群中的节点，请在复用之前[清理节点](../../../../how-to-guides/new-user-guides/manage-clusters/clean-cluster-nodes.md)。如果你重复使用尚未清理的节点，则集群配置可能会失败。 + +根据[安装要求](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)和[生产就绪集群的检查清单](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md)配置主机。 + +如果你使用 Amazon EC2 作为主机，并希望使用[双栈 (dual-stack)](https://kubernetes.io/docs/concepts/services-networking/dual-stack/) 功能，则需要满足配置主机的其他[要求](https://rancher.com/docs/rke//latest/en/config-options/dual-stack#requirements)。 + +### 2. 创建自定义集群 + +1. 点击 **☰ > 集群管理**。 +1. 在**集群**页面上，单击**创建**。 +1. 单击**自定义**。 +1. 输入**集群名称**。 +1. 在**集群配置**中，选择 Kubernetes 版本、要使用的网络提供商，以及是否启用项目网络隔离。要查看更多集群选项，请单击**显示高级选项**。 + + :::note 使用 Windows 主机作为 Kubernetes Worker 节点？ + + - 请参阅[启用 Windows 支持选项](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)。 + - 支持 Windows 集群的唯一网络插件是 Flannel。 + + ::: + + :::note Amazon EC2 上的双栈： + + 如果你使用 Amazon EC2 作为主机，并希望使用[双栈 (dual-stack)](https://kubernetes.io/docs/concepts/services-networking/dual-stack/) 功能，则需要满足配置 RKE 的其他[要求](https://rancher.com/docs/rke//latest/en/config-options/dual-stack#requirements)。 + + ::: + +6. 点击**下一步**。 + +4. 使用**成员角色**为集群配置用户授权。点击**添加成员**添加可以访问集群的用户。使用**角色**下拉菜单为每个用户设置权限。 + +7. 从**节点角色**中，选择要由集群节点充当的角色。你必须为 `etcd`、`worker` 和 `controlplane` 角色配置至少一个节点。自定义集群需要所有三个角色才能完成配置。有关角色的详细信息，请参阅[本节](../../../kubernetes-concepts.md#kubernetes-集群中节点的角色)。 + +:::note + +- 使用 Windows 主机作为 Kubernetes Worker 节点？请参阅[本节](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)。 +- 裸机服务器提醒：如果你想将裸机服务器专用于每个角色，则必须为每个角色配置一个裸机服务器（即配置多个裸机服务器）。 + +::: + +8. **可选**：点击**[显示高级选项](../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md)**来指定注册节点时使用的 IP 地址，覆盖节点的主机名，或将[标签](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)或[污点](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)添加到节点。 + +9. 将屏幕上显示的命令复制到剪贴板。 + +10. 使用你惯用的 shell（例如 PuTTy 或远程终端）登录到你的 Linux 主机。粘贴剪贴板的命令并运行。 + +:::note + +如果要将特定主机专用于特定节点角色，请重复步骤 7-10。根据需要多次重复这些步骤。 + +::: + +11. 在 Linux 主机上运行完命令后，单击**完成**。 + +**结果：** + +集群已被创建，集群的状态是**配置中**。Rancher 已在启动你的集群。 + +当集群状态变为 **Active** 后，你可访问集群。 + +**Active** 状态的集群会分配到两个项目： + +- `Default`：包含 `default` 命名空间 +- `System`：包含 `cattle-system`，`ingress-nginx`，`kube-public` 和 `kube-system` 命名空间。 + + +### 3. 仅限亚马逊：标签资源 + +如果你已将集群配置为使用 Amazon 作为**云提供商**，请使用集群 ID 标记你的 AWS 资源。 + +[Amazon 文档：标记你的 Amazon EC2 资源](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) + +:::note + +你可以使用 Amazon EC2 实例，而无需在 Kubernetes 中配置云提供商。如果你想使用特定的 Kubernetes 云提供商功能，配置云提供商即可。如需更多信息，请参阅 [Kubernetes 云提供商](https://github.com/kubernetes/website/blob/release-1.18/content/en/docs/concepts/cluster-administration/cloud-providers.md)。 + +::: + +以下资源需要使用 `ClusterID` 进行标记： + +- **Nodes**：Rancher 中添加的所有主机。 +- **Subnet**：集群使用的子网。 +- **Security Group**：用于你的集群的安全组。 + +:::note + +不要标记多个安全组。创建 Elastic Load Balancer 时，标记多个组会导致错误。 + +::: + +应该使用的标签是： + +``` +Key=kubernetes.io/cluster/, Value=owned +``` + +`` 可以是你选择的任何字符串。但是，必须在你标记的每个资源上使用相同的字符串。将值设置为 `owned` 会通知集群所有带有 `` 标记的资源都由该集群拥有和管理。 + +如果你在集群之间共享资源，你可以将标签更改为： + +``` +Key=kubernetes.io/cluster/CLUSTERID, Value=shared +``` + +## 可选的后续步骤 + +创建集群后，你可以通过 Rancher UI 访问集群。最佳实践建议你设置以下访问集群的备用方式： + +- **通过 kubectl CLI 访问你的集群**：按照[这些步骤](../../../../how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#在工作站使用-kubectl-访问集群)在你的工作站上使用 kubectl 访问集群。在这种情况下，你将通过 Rancher Server 的认证代理进行认证，然后 Rancher 会让你连接到下游集群。此方法允许你在没有 Rancher UI 的情况下管理集群。 +- **通过 kubectl CLI 使用授权的集群端点访问你的集群**：按照[这些步骤](../../../../how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#直接使用下游集群进行身份验证)直接使用 kubectl 访问集群，而无需通过 Rancher 进行认证。我们建议设置此替代方法来访问集群，以便在无法连接到 Rancher 时访问集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md new file mode 100644 index 00000000000..edc2686bb64 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md @@ -0,0 +1,20 @@ +--- +title: 架构 +--- + + + + + +本章节重点介绍 [Rancher Server 及其组件](rancher-server-and-components.md) 以及 [Rancher 如何与下游 Kubernetes 集群通信](communicating-with-downstream-user-clusters.md) +有关安装 Rancher 的不同方式的信息，请参见[安装选项概述](../../getting-started/installation-and-upgrade/installation-and-upgrade.md#安装方式概述)。 + +有关 Rancher API Server 的主要功能，请参见[概述](../../getting-started/overview.md#rancher-api-server-的功能)。 + +有关如何为 Rancher Server 设置底层基础架构，请参见[架构推荐](architecture-recommendations.md)。 + +:::note + +本节默认你已对 Docker 和 Kubernetes 有一定的了解。如果你需要了解 Kubernetes 组件如何协作，请参见 [Kubernetes 概念](../kubernetes-concepts.md)。 + +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/single-node-rancher-in-docker/single-node-rancher-in-docker.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/single-node-rancher-in-docker/single-node-rancher-in-docker.md new file mode 100644 index 00000000000..e272daa1ca4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/single-node-rancher-in-docker/single-node-rancher-in-docker.md @@ -0,0 +1,9 @@ +--- +title: Docker 中的单节点 Rancher +--- + + + + + +以下文档将讨论 Docker 安装的 [HTTP 代理配置](http-proxy-configuration.md)和[高级选项](advanced-options.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters.md new file mode 100644 index 00000000000..12925c800a9 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters.md @@ -0,0 +1,140 @@ +--- +title: 下游集群开启 API 审计日志 +--- + + + + + +Kubernetes 审计提供了由 Kube-apiserver 执行的与安全相关的、按时间顺序排列的集群审计记录。Kube API 会在请求执行的每个阶段都生成一个事件，然后根据策略进行预处理并保存，审计策略配置了要记录的内容。 + +你可能希望将审计日志配置为遵守互联网安全中心 (CIS) Kubernetes 基准控制的一部分。 + +有关配置的详细信息，请参阅 [Kubernetes 官方文档](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/)。 + + + + +:::note + +Rancher v2.7.2 及以上版本提供此功能。 + +::: + +首先，你需要创建一个 Secret 或 ConfigMap，用于配置审计策略。 + +Secret 或 ConfigMap 必须满足以下两个要求： + +1. 必须位于 Cluster 对象所在的 `fleet-default` 命名空间中。 +2. 它必须具有注释 `rke.cattle.io/object-authorized-for-clusters: cluster-name1,cluster-name2`，以允许目标集群使用它。 + +:::tip + +Rancher Dashboard 提供了易于使用的表单页面用于创建 Secret 或 ConfigMap。 + +::: + +例子： + +```yaml +apiVersion: v1 +data: + audit-policy: >- + IyBMb2cgYWxsIHJlcXVlc3RzIGF0IHRoZSBNZXRhZGF0YSBsZXZlbC4KYXBpVmVyc2lvbjogYXVkaXQuazhzLmlvL3YxCmtpbmQ6IFBvbGljeQpydWxlczoKLSBsZXZlbDogTWV0YWRhdGE= +kind: Secret +metadata: + annotations: + rke.cattle.io/object-authorized-for-clusters: cluster1 + name: name1 + namespace: fleet-default +``` + +可以通过编辑集群 YAML 的 `machineSelectorFiles` 和 `machineGlobalConfig` 字段来启用和配置审计日志。 + +例子： + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +spec: + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - audit-policy-file=/dev-audit-policy.yaml + - audit-log-path=/dev-audit.logs + machineSelectorFiles: + - fileSources: + - configMap: + name: '' + secret: + items: + - key: audit-policy + path: /dev-audit-policy.yaml + name: dev-audit-policy + machineLabelSelector: + matchLabels: + rke.cattle.io/control-plane-role: 'true' +``` + +有关集群配置的更多信息，请参阅 REK2 或 K3s 集群配置参考页。 + + + + + +可通过编辑集群 YAML 来启用和配置审计日志。 + +在启用审计日志后，将使用 RKE1 的默认值。 + +```yaml +# +# Rancher Config +# +rancher_kubernetes_engine_config: + services: + kube-api: + audit_log: + enabled: true +``` + +你还可以自定义审计日志配置。 + +```yaml +# +# Rancher Config +# +rancher_kubernetes_engine_config: + services: + kube-api: + audit_log: + enabled: true + configuration: + max_age: 6 + max_backup: 6 + max_size: 110 + path: /var/log/kube-audit/audit-log.json + format: json + policy: + apiVersion: audit.k8s.io/v1 # 这里必须填写 + kind: Policy + omitStages: + - "RequestReceived" + rules: + # Log pod changes at RequestResponse level + - level: RequestResponse + resources: + - group: "" + # Resource "pods" doesn't match requests to any subresource of pods, + # which is consistent with the RBAC policy. + resources: ["pods"] + # Log "pods/log", "pods/status" at Metadata level + - level: Metadata + resources: + - group: "" + resources: ["pods/log", "pods/status"] +``` + +配置详情请参考 [RKE1 官方文档](https://rke.docs.rancher.com/config-options/audit-log)。 + + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/backup-restore-configuration/backup-restore-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/backup-restore-configuration/backup-restore-configuration.md new file mode 100644 index 00000000000..6fbc1f202af --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/backup-restore-configuration/backup-restore-configuration.md @@ -0,0 +1,12 @@ +--- +title: Rancher 备份配置参考 +--- + + + + + +- [备份配置](backup-configuration.md) +- [还原配置](restore-configuration.md) +- [存储位置配置](storage-configuration.md) +- [Backup 和 Restore 自定义资源示例](examples.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/best-practices.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/best-practices.md new file mode 100644 index 00000000000..11f4ee82e41 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/best-practices.md @@ -0,0 +1,21 @@ +--- +title: 最佳实践 +--- + + + + + +本节介绍 Rancher 实现的最佳实践，其中包括对 Kubernetes、Docker、容器等技术的使用建议。最佳实践旨在利用 Rancher 及其客户的运营经验，助你更好地实现 Rancher。 + +如果你对用例的实际应用有任何疑问，请联系客户成功经理或支持中心。 + +你可以在左侧导航栏快速找到管理和部署 Rancher Server 的最佳实践。 + +如需查看更多最佳实践指南，请参见： + +- [安全类文档](../rancher-security/rancher-security.md) +- [Rancher 博客](https://www.suse.com/c/rancherblog/) +- [Rancher 论坛](https://forums.rancher.com/) +- [Rancher 用户的 Slack 群组](https://slack.rancher.io/) +- [B 站](https://space.bilibili.com/430496045/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md new file mode 100644 index 00000000000..e12f1db2498 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md @@ -0,0 +1,23 @@ +--- +title: Rancher 管理集群的最佳实践 +--- + + + + + +### Logging + +有关集群级别日志和应用日志的建议，请参见 [Logging 最佳实践](logging-best-practices.md)。 + +### Monitoring + +配置合理的监控和告警规则对于安全、可靠地运行生产环境中的工作负载至关重要。有关更多建议，请参阅[最佳实践](monitoring-best-practices.md)。 + +### 设置容器的技巧 + +配置良好的容器可以极大地提高环境的整体性能和安全性。有关容器设置的建议，请参见[设置容器的技巧](tips-to-set-up-containers.md)。 + +### Rancher 管理 vSphere 集群的最佳实践 + +[Rancher 管理 vSphere 集群的最佳实践](rancher-managed-clusters-in-vsphere.md)概述了在 vSphere 环境中配置下游 Rancher 集群的参考架构，以及 VMware 记录的标准 vSphere 最佳实践。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/rancher-server/rancher-server.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/rancher-server/rancher-server.md new file mode 100644 index 00000000000..d297b395a0a --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/rancher-server/rancher-server.md @@ -0,0 +1,21 @@ +--- +title: Rancher Server 的最佳实践 +--- + + + + + +本指南介绍了让 Rancher 管理下游 Kubernetes 集群的 Rancher Server 运行建议。 + +### 推荐的架构和基础设施 + +有关在高可用 Kubernetes 集群上设置 Rancher Server 的通用建议，请参见[本指南](tips-for-running-rancher.md)。 + +### 部署策略 + +[本指南](rancher-deployment-strategy.md)旨在帮助你选择部署策略（区域部署/中心辐射型部署），来让 Rancher Server 更好地管理下游 Kubernetes 集群。 + +### 在 vSphere 环境中安装 Rancher + +[本指南](on-premises-rancher-in-vsphere.md)介绍了在 vSphere 环境中安装 Rancher 的参考架构，以及 VMware 记录的标准 vSphere 最佳实践。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md new file mode 100644 index 00000000000..552e79ec76b --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md @@ -0,0 +1,117 @@ +--- +title: Rancher 大规模部署的调优和最佳实践 +--- + + + + + +本指南介绍了扩容 Rancher 相关场景的最佳实践和调优方法。随着系统规模的不断增长，性能自然会降低，但是可以采取一些步骤将 Rancher 的负载最小化，以优化 Rancher 管理较大基础设施的能力。 + +## 优化 Rancher 性能 + +* 及时升级 Rancher 的 Patch 更新。我们会不断的对 Rancher 进行性能增强和错误修复，最新的 Rancher 版本包含了基于开发人员和用户反馈的所有性能和稳定性的优化改进。 +* 逐渐的执行扩容操作，在操作的过程中观察并监控任何变化，在性能刚出现异常时发现问题并修复，避免其他干扰因素。 +* 尽可能地减少上游 Rancher 集群和下游集群之间的网络延时。需要注意的是，排除掉其他因素，延时与地理距离成正比，如果你的集群或节点分布在全球各地，请考虑改为安装多个 Rancher。 + +## 最小化上游集群的负载 + +在 Rancher 扩容时，一个典型的性能瓶颈是上游（local）Kubernetes 集群的资源增长。上游集群包含所有下游集群的数据信息，许多对下游集群的操作会在上游集群中创建新的对象，并需要在上游集群中进行计算处理。 + +### 最大限度地减少上游集群上的第三方软件 + +大规模运行 Rancher 可能会给内部 Kubernetes 组件（例如 `etcd` 或 `kubeapiserver`）带来大量负载。如果第三方软件干扰了这些组件或 Rancher 的性能，则可能会出现性能问题。 + +每个第三方软件都存在干扰性能的风险。为了防止上游集群出现性能问题，你应该避免运行除 Kubernetes 系统组件和 Rancher 本身之外的任何应用程序或组件。 + +以下类别的软件通常不会干扰 Rancher 或 Kubernetes 系统性能： + * Rancher 内部组件，例如 Fleet + * Rancher 扩展 + * 集群 API 组件 + * CNI 网络插件 + * 云控制器管理器 + * 观察和监控工具（`prometheus-rancher-exporter` 除外） + +除此之外，已发现以下软件会在 Rancher 扩容时影响性能： + * [CrossPlane](https://www.crossplane.io/) + * [Argo CD](https://argoproj.github.io/cd/) + * [Flux](https://fluxcd.io/) + * [prometheus-rancher-exporter](https://github.com/David-VTUK/prometheus-rancher-exporter)（请参阅 [issue 33](https://github.com/David-VTUK/prometheus-rancher-exporter/issues/33)） + +### 管理你的 Object 计数 + +Etcd 是 Kubernetes 和 Rancher 的后端数据库，该数据库可能会遇到单个 Kubernetes 资源类型数量的限制，确切的限制因素各不相同，取决于诸多因素。经验表明，一旦单个资源类型的对象数量超过 60000，通常会出现性能问题，通常该资源类型是 RoleBinding 对象。 + +这在 Rancher 中很常见，因为许多操作会在上游集群中创建新的 RoleBinding 对象。 + +你可以通过以下方法减少上游集群的 `RoleBinding` 数量： +* 减少使用 [Restricted Admin（受限管理员）](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md#受限管理员) 角色，改为使用其他角色。 +* 如果你使用了 [外部认证](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md)，配置其按组分配角色。 +* 仅在必要情况下将用户添加到集群和项目中。 +* 移除不需要的集群和项目。 +* 仅在必要情况下使用自定义角色。 +* 在自定义角色中，尽可能的减少规则数量。 +* 避免为用户添加多余的角色。 +* 考虑使用数量更少，但性能更强大的集群。 +* Kubernetes 权限始终是 “附加”（允许列表）而不是“减去”（拒绝列表）。尽量减少允许访问集群、项目或命名空间等方面的配置，因为这将导致创建大量的 RoleBinding 对象。 +* 实验一下，创建新项目或集群后，在你的特定用例场景是否表现出更少的 RoleBinding 的效果。 + +### RoleBinding 计数估计 + +预测给定的配置将创建多少个 RoleBinding 对象很复杂，但是以下注意事项可以提供粗略的估算： +* 对于最小的数量估计，请使用公式 `32C + U + 2UaC + 8P + 5Pa`。 + * `C` 是集群数量。 + * `U` 是用户数量。 + * `Ua` 是集群中具有成员资格的用户的平均数量。 + * `P` 是项目总数。 + * `Pa` 是项目中具有成员资格的平均用户数。 +* 受限管理员角色（Restricted Admin）遵循不同的公式，因为具有此角色的每个用户都会额外产生至少 `7C + 2P + 2` 个 `RoleBinding` 对象。 +* * `RoleBinding` 的数量会随着集群、项目和用户的数量线性增加。 + +### 使用新的应用代替旧版应用 + +Rancher 使用两个 Kubernetes 应用程序资源：`apps.projects.cattle.io` 和 `apps.cattle.cattle.io`。以`apps.projects.cattle.io` 为代表的旧版应用程序在低版本的 Cluster Manager UI 中引入，现已弃用。新的应用程序（由 `apps.catalog.cattle.io` 表示）可以在 Cluster Explorer UI 中安装部署。新的 `apps.cattle.cattle.io` 应用程序数据保存在下游集群中，这可以减少上游集群中的资源数量。 + +你应当删除 Cluster Manager UI 中遗留的所有旧版应用程序，并将其替换为 Cluster Explorer UI 中的应用程序。只在 Cluster Explorer UI 中创建任何新应用程序。 + +### 使用授权集群端点 (ACE) + +[授权集群端点](../../../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters.md#4-授权集群端点) (ACE) 提供了 Rancher 部署的 RKE、RKE2 和 K3s 集群的 Kubernetes API 访问。启用后，ACE 会为生成的 `kubeconfig` 文件配置直接访问下游集群 Endpoint，从而绕过 Rancher 代理。在可以直接访问下游集群 Kubernetes API 的场景下，可以减少 Rancher 负载。有关更多信息，请参阅[授权集群端点](../../../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters.md#4-授权集群端点)配置说明。 + +### 减少 Event Handler 执行 + +Rancher 的大部分逻辑发生在 Event Handler 上。每当资源对象产生更新或 Rancher 启动时，这些资源对应的 Event Handler 都会执行。除此之外，它们会每隔 15 小时在 Rancher 计划的同步缓存时再运行一次，这可能会导致 Rancher 运行过程中出现大量性能消耗。可使用 `CATTLE_SYNC_ONLY_CHANGED_OBJECTS` 环境变量禁用计划的 Handler 处理程序执行。如果每 15 小时出现一次性能资源高峰，此设置会有所帮助。 + +`CATTLE_SYNC_ONLY_CHANGED_OBJECTS` 的值可被设置为以下内容，以逗号分隔。这些值代表了处理程序的种类，将处理程序添加到该变量会禁止其在定期的缓存重新同步过程中运行。 + +* `mgmt`：在 Rancher 节点上运行的 Management 管理控制器。 +* `user`：所有集群运行的用户控制器。其中一部分与 Management 管理控制器运行在同一节点上，而另一部分运行在下游集群中，该选项针对的是前者。 +* `scaled`：每个 Rancher 节点上运行的规模控制器。因规模控制器负责关键功能，应当避免设置此值。如果设置可能会破坏集群稳定。 + +简而言之，如果你发现 CPU 使用率每 15 小时出现一次峰值，请将 `CATTLE_SYNC_ONLY_CHANGED_OBJECTS` 环境变量添加到 Rancher Deployment 中（添加至 `spec.containers.env` 列表），其值为 `mgmt,user`。 + +## Rancher 之外的优化 + +集群底层自身的配置也是影响性能的重要因素。如果上游集群存在错误配置，会带来 Rancher 软件所无法解决的性能瓶颈。 + +### 使用 RKE2 直接管理上游集群节点 + +由于 Rancher 对上游集群的要求非常高，尤其是在大规模部署场景，你需要拥有上游集群和节点的所有管理员权限，要找出资源消耗过高的根本原因，请使用标准的 Linux 故障排除工具，这有助于区分是 Rancher、Kubernetes 还是操作系统组件出现的问题。 + +尽管托管 Kubernetes 服务使部署和运行 Kubernetes 集群变得更加容易，但在大规模场景中，不鼓励将其用于上游集群。 托管 Kubernetes 服务通常会限制对单个节点和服务配置的访问。 + +建议在大规模用例场景中使用 RKE2 集群。 + +### 及时更新 Kubernetes 版本 + +你应当及时更新上游集群的 Kubernetes 版本，以确保你的集群具备最新的性能增强和问题修复。 + +### 优化 Etcd + +Etcd 是 Kubernetes 和 Rancher 的后端数据库，在 Rancher 性能中扮演重要的角色。 + +[Etcd 性能](https://etcd.io/docs/v3.4/op-guide/performance/)的两个主要瓶颈是磁盘和网络速度。Etcd 应当在具有高速网络和高读写速度 (IOPS) SSD 硬盘的专用节点上运行。有关 etcd 性能的更多信息，请参阅 [etcd 性能缓慢（性能测试和优化）](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装进行 etcd 调优](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md)。有关磁盘的信息可以在[安装要求](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#磁盘)中找到。 + +根据 etcd 的[复制机制](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size)，建议在三个节点上运行 etcd，运行在更多的节点上反而会降低速度。 + +Etcd 性能也会受节点之间的网络延迟影响，因此 etcd 节点应与 Rancher 节点部署在一起。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/cluster-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/cluster-configuration.md new file mode 100644 index 00000000000..9bc1db8d00b --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/cluster-configuration.md @@ -0,0 +1,32 @@ +--- +title: 集群配置 +--- + + + + + +使用 Rancher 配置 Kubernetes 集群后，你仍然可以编辑集群的选项和设置。 + +有关编辑集群成员资格的信息，请转至[此页面](../../how-to-guides/new-user-guides/manage-clusters/access-clusters/add-users-to-clusters.md)。 + +### 集群配置参考 + +集群配置选项取决于 Kubernetes 集群的类型： + +- [RKE 集群配置](rancher-server-configuration/rke1-cluster-configuration.md) +- [RKE2 集群配置](rancher-server-configuration/rke2-cluster-configuration.md) +- [K3s 集群配置](rancher-server-configuration/k3s-cluster-configuration.md) +- [EKS 集群配置](rancher-server-configuration/eks-cluster-configuration.md) +- [GKE 集群配置](gke-cluster-configuration.md) +- [AKS 集群配置](rancher-server-configuration/aks-cluster-configuration.md) + +### 不同类型集群的管理功能 + +对于已有集群而言，可提供的选项和设置取决于你配置集群的方法。 + +下表总结了每一种类型的集群和对应的可编辑的选项和设置： + +import ClusterCapabilitiesTable from '../../shared-files/_cluster-capabilities-table.md'; + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/downstream-cluster-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/downstream-cluster-configuration.md new file mode 100644 index 00000000000..591285c378e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/downstream-cluster-configuration.md @@ -0,0 +1,9 @@ +--- +title: 下游集群配置 +--- + + + + + +以下文档将讨论[节点模板配置](./node-template-configuration.md)和[主机配置](./machine-configuration.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/machine-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/machine-configuration.md new file mode 100644 index 00000000000..7669ec23377 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/machine-configuration.md @@ -0,0 +1,9 @@ +--- +title: 主机配置 +--- + + + + + +主机配置指的是如何将资源分配给虚拟机。请参阅 [Amazon EC2](amazon-ec2.md)、[DigitalOcean](digitalocean.md) 和 [Azure](azure.md) 的文档以了解更多信息。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/node-template-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/node-template-configuration.md new file mode 100644 index 00000000000..35779a5d3ae --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/node-template-configuration.md @@ -0,0 +1,9 @@ +--- +title: 节点模板配置 +--- + + + + + +要了解节点模板配置，请参阅[EC2 节点模板配置](amazon-ec2.md)、[DigitalOcean 节点模板配置](digitalocean.md)、[Azure 节点模板配置](azure.md)、[vSphere 节点模板配置](vsphere.md)和 [Nutanix 节点模板配置](nutanix.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md new file mode 100644 index 00000000000..80081aefe75 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md @@ -0,0 +1,324 @@ +--- +title: GKE 集群配置参考 +--- + + + + + +## Rancher 2.6 变更 + +- 支持额外的配置选项： + - 项目网络隔离 + - 网络标签 + +## 集群位置 + +| 值 | 描述 | +|--------|--------------| +| 位置类型 | 地区 (zone) 或区域 (region)。借助 GKE，你可以根据工作负载的可用性要求和预算创建一个量身定制的集群。默认情况下，集群的节点在单个计算区域中运行。选择多个区域时，集群的节点将跨越多个计算区域，而 controlplane 则只位于单个区域中。区域集群也增加了 controlplane 的可用性。有关选择集群可用性类型的帮助，请参阅[这些文档](https://cloud.google.com/kubernetes-engine/docs/best-practices/scalability#choosing_a_regional_or_zonal_control_plane)。 | +| 地区 | 计算引擎中的每个区域都包含多地区。有关可用区域和可用区的更多信息，请参阅[这些文档](https://cloud.google.com/compute/docs/regions-zones#available)。 | +| 其他地区 | 对于地区性集群，你可以选择其他地区来创建[多地区集群](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#multi-zonal_clusters)。 | +| 区域 | 对于[区域性集群](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#regional_clusters)，你可以选择一个区域。有关可用区域和可用区的更多信息，请参阅[本节](https://cloud.google.com/compute/docs/regions-zones#available)。地区名称的前面部分是区域的名称。 | + +## 集群选项 + +### Kubernetes 版本 + +_可变：是_ + +有关 GKE Kubernetes 版本的更多信息，请参阅[这些文档](https://cloud.google.com/kubernetes-engine/versioning)。 + +### 容器地址范围 + +_可变：否_ + +集群中 Pod 的 IP 地址范围。必须是有效的 CIDR 范围，例如 10.42.0.0/16。如果未指定，则会自动从 10.0.0.0/8 中选择一个随机范围，并排除已分配给 VM、其他集群或路由的范围。自动选择的范围可能与预留的 IP 地址、动态路由或与集群对等的 VPC 中的路由发生冲突。 + +### 网络 + +_可变：否_ + +集群连接的 Compute Engine 网络。将使用此网络创建路由和防火墙。如果使用[共享 VPC](https://cloud.google.com/vpc/docs/shared-vpc)，与你的项目共享的 VPC 网络将显示在此处。你将可以在此字段中进行选择。有关详细信息，请参阅[此页面](https://cloud.google.com/vpc/docs/vpc#vpc_networks_and_subnets)。 + +### 节点子网/子网 + +_可变：否_ + +集群连接到的 Compute Engine 子网。该子网必须属于**网络**字段中指定的网络。选择一个现有的子网，或选择“自动创建子网”来自动创建一个子网。如果不使用现有网络，则需要使用**子网名称**来生成一个。如果使用[共享 VPC](https://cloud.google.com/vpc/docs/shared-vpc)，与你的项目共享的 VPC 子网将显示在此处。如果使用共享 VPC 网络，则无法选择“自动创建子网”。如需更多信息，请参阅[此页面](https://cloud.google.com/vpc/docs/vpc#vpc_networks_and_subnets)。 + +### 子网名称 + +_可变：否_ + +使用提供的名称自动创建子网。如果为**节点子网**或**子网**选择了“自动创建子网”，则为必填。有关子网的更多信息，请参阅[此页面](https://cloud.google.com/vpc/docs/vpc#vpc_networks_and_subnets)。 + +### IP 别名 + +_可变：否_ + +启用[别名 IP](https://cloud.google.com/vpc/docs/alias-ip)。这将启用 VPC 原生流量路由。如果使用[共享 VPC](https://cloud.google.com/vpc/docs/shared-vpc)，则为必填。 + +### 网络策略 + +_可变：是_ + +在集群上启用的网络策略。网络策略定义了集群中 pod 和 service 之间可以发生的通信级别。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/how-to/network-policy)。 + +### 项目网络隔离 + +_可变：是_ + +选择启用或禁用项目间通信。请注意，如果启用**项目网络隔离**，则将自动启用**网络策略**和**网络策略配置**，反之则不然。 + +### 节点 IPv4 CIDR 块 + +_可变：否_ + +此集群中实例 IP 的 IP 地址范围。如果为**节点子网**或**子网**选择了“自动创建子网”，则可以进行设置。必须是有效的 CIDR 范围，例如 10.96.0.0/14。有关如何确定 IP 地址范围的详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing)。 + +### 集群次要范围名称 + +_可变：否_ + +Pod IP 地址的现有次要范围的名称。如果选中，将自动填充**集群 Pod 地址范围**。如果使用共享 VPC 网络，则为必填。 + +### 集群 Pod 地址范围 + +_可变：否_ + +分配给集群中 pod 的 IP 地址范围。必须是有效的 CIDR 范围，例如 10.96.0.0/11。如果未提供，将自动创建。如果使用共享 VPC 网络，则必须提供。有关如何确定 pod 的 IP 地址范围的更多信息，请参阅[本节](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing_secondary_range_pods)。 + +### Service 次要范围名称 + +_可变：否_ + +Service IP 地址的现有次要范围的名称。如果选中，将自动填充 **Service 地址范围**。如果使用共享 VPC 网络，则为必填。 + +### Service 地址范围 + +_可变：否_ + +分配给集群中 Service 的地址范围。必须是有效的 CIDR 范围，例如 10.94.0.0/18。如果未提供，将自动创建。如果使用共享 VPC 网络，则必须提供。有关如何确定 Service 的 IP 地址范围的详细信息，请参阅[本节](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing_secondary_range_svcs)。 + +### 私有集群 + +_可变：否_ + +:::caution + +私有集群需要在 Rancher 之外进行额外的规划和配置。请参阅[私有集群指南](gke-private-clusters.md)。 + +::: + +仅分配节点内部 IP 地址。除非在 GCP 中执行了额外的联网步骤，否则私有集群节点无法访问公共互联网。 + +### 启用私有端点 + +:::caution + +私有集群需要在 Rancher 之外进行额外的规划和配置。请参阅[私有集群指南](gke-private-clusters.md)。 + +::: + +_可变：否_ + +锁定对 controlplane 端点的外部访问。仅当**私有集群**也被选中时可用。如果选中，并且 Rancher 无法直接访问集群所在的虚拟私有云网络，Rancher 将提供在集群上运行的注册命令，以使 Rancher 能够连接到集群。 + +### 主 IPV4 CIDR 块 + +_可变：否_ + +controlplane VPC 的 IP 范围。 + +### 主授权网络 + +_可变：是_ + +启用 controlplane 授权网络，以阻止不受信任的非 GCP 源 IP 通过 HTTPS 访问 Kubernetes master。如果选择，则可以添加额外的授权网络。如果集群是使用公共端点创建的，则此选项可用于将公共端点的访问锁定到特定网络（例如运行 Rancher 服务的网络）。如果集群只有一个私有端点，则需要此设置。 + +## 其他选项 + +### 集群插件 + +其他 Kubernetes 集群组件。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters#Cluster.AddonsConfig)。 + +#### 水平 Pod 自动缩放 + +_可变：是_ + +Horizo​​ntal Pod Autoscaler 通过自动增加或减少 Pod 的数量来调整 Kubernetes 工作负载，从而响应工作负载的 CPU 或内存消耗，以及 Kubernetes 内部报告的自定义指标或集群外部设置的指标。详情请参见[本页面](https://cloud.google.com/kubernetes-engine/docs/concepts/horizontalpodautoscaler)。 + +#### HTTP (L7) 负载均衡 + +_可变：是_ + +HTTP (L7) 负载均衡将 HTTP 和 HTTPS 流量分配到托管在 GKE 上的后端。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer)。 + +#### 网络策略配置（仅限 master） + +_可变：是_ + +NetworkPolicy 的配置。仅跟踪 master 节点上是否启用了插件，不跟踪是否为节点启用了网络策略。 + +### 集群特征（Alpha 功能） + +_可变：否_ + +打开集群的所有 Kubernetes alpha API 组和功能。启用后，集群无法升级，并且会在 30 天后自动删除。由于 GKE SLA 未支持 alpha 集群，因此不建议将 Alpha 集群用于生产环境。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/concepts/alpha-clusters)。 + +### Logging 服务 + +_可变：是_ + +集群用于写入日志的日志管理服务。要么使用 [Cloud Logging](https://cloud.google.com/logging)，要么不使用日志管理服务（不会从集群中导出日志）。 + +### 监控服务 + +_可变：是_ + +集群用于写入指标的监控服务。要么使用 [Cloud Monitoring](https://cloud.google.com/monitoring)，要么不使用集群监控服务（不会从集群中导出指标）。 + + +### 维护窗口 + +_可变：是_ + +设置时长 4 小时的维护窗口的开始时间。使用 HH:MM 格式在 UTC 时区中指定时间。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/concepts/maintenance-windows-and-exclusions)。 + +## 节点池 + +在此部分中，输入描述节点池中每个节点的配置的详细信息。 + +### Kubernetes 版本 + +_可变：是_ + +节点池中每个节点的 Kubernetes 版本。有关 GKE Kubernetes 版本的更多信息，请参阅[这些文档](https://cloud.google.com/kubernetes-engine/versioning)。 + +### 镜像类型 + +_可变：是_ + +节点操作系统镜像。有关 GKE 为每个操作系统提供的节点镜像选项，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#available_node_images)。 + +:::note + +默认选项是 “Container-Optimized OS with Docker”。GCP Container-Optimized OS 上的只读文件系统与 Rancher 中的 [legacy logging](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/cluster-logging/cluster-logging.md) 实现不兼容。如果你需要使用旧版日志管理功能，请选择 “Ubuntu with Docker” 或 “Ubuntu with Containerd”。[current logging feature](../../../../integrations-in-rancher/logging/logging.md) 与 Container-Optimized OS 镜像兼容。 + +::: + +:::note + +如果节点池镜像类型选择 “Windows Long Term Service Channel” 或 “Windows Semi-Annual Channel”，还必须至少添加一个 Container-Optimized OS 或 Ubuntu 节点池。 + +::: + +### 主机类型 + +_可变：否_ + +节点实例可用的虚拟化硬件资源。有关 Google Cloud 主机类型的详细信息，请参阅[此页面](https://cloud.google.com/compute/docs/machine-types#machine_types)。 + +### 根磁盘类型 + +_可变：否_ + +标准永久性磁盘由标准磁盘驱动器 (HDD) 支持，而 SSD 永久性磁盘由固态硬盘 (SSD) 支持。有关详细信息，请参阅[本节](https://cloud.google.com/compute/docs/disks)。 + +### 本地 SSD 磁盘 + +_可变：否_ + +配置每个节点的本地 SSD 磁盘存储（以 GB 为单位）。本地 SSD 物理连接到托管你的 VM 实例的服务器。与标准永久性磁盘或 SSD 永久性磁盘相比，本地 SSD 具有更高的吞吐量和更低的延迟。存储在本地 SSD 上的数据只会保留到实例停止或删除。有关详细信息，请参阅[本节](https://cloud.google.com/compute/docs/disks#localssds)。 + +### 抢占式节点（beta） + +_可变：否_ + +抢占式节点也称为抢占式虚拟机。通常是最长持续 24 小时的 Compute Engine 虚拟机实例，不提供可用性保证。详情请参见[本页面](https://cloud.google.com/kubernetes-engine/docs/how-to/preemptible-vms)。 + +### 污点 + +_可变：否_ + +将污点应用于节点时，仅允许容忍该污点的 Pod 在该节点上运行。在 GKE 集群中，你可以将污点应用到节点池，这会将污点应用到池中的所有节点。 + +### 节点标签 + +_可变：否_ + +你可以将标签应用到节点池，这会将标签应用到池中的所有节点。 + +无效标签会阻止升级，或阻止 Rancher 启动。有关标签语法的详细信息，请参阅 [Kubernetes 文档](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set)。 + +### 网络标签 + +_可变：否_ + +你可以将网络标签添加到节点池以制定防火墙规则和子网之间的路由。标签将应用于池中的所有节点。 + +有关标签语法和要求的详细信息，请参阅 [Kubernetes 文档](https://cloud.google.com/vpc/docs/add-remove-network-tags)。 + +## 组详细信息 + +在此部分中，输入描述节点池的详细信息。 + +### 名称 + +_可变：否_ + +输入节点池的名称。 + +### 初始节点数 + +_可变：是_ + +节点池中初始节点数的整数。 + +### 每个节点的最大 Pod 数量 + +_可变：否_ + +GKE 的硬性限制是每个节点 110 个 Pod。有关 Kubernetes 限制的更多信息，请参阅[本节](https://cloud.google.com/kubernetes-engine/docs/best-practices/scalability#dimension_limits)。 + +### 自动缩放 + +_可变：是_ + +节点池自动缩放会根据工作负载的需求动态创建或删除节点。详情请参见[本页面](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler)。 + +### 自动修复 + +_可变：是_ + +GKE 的节点自动修复功能可帮助你将集群中的节点保持在健康的运行状态。启用后，GKE 会定期检查集群中每个节点的运行状况。如果某个节点在较长时间段内连续未通过健康检查，GKE 会为该节点启动修复过程。有关详细信息，请参阅[自动修复节点](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair)。 + +### 自动升级 + +_可变：是_ + +启用后，当你的 controlplane [按照你的需求更新](https://cloud.google.com/kubernetes-engine/upgrades#automatic_cp_upgrades)时，自动升级功能会使集群中的节点与集群 controlplane（master）版本保持同步。有关自动升级节点的更多信息，参见[此页面。](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades) + +### 访问范围 + +_可变：否_ + +设置访问范围是为你的节点指定权限的旧版方法。 + +- **允许默认访问**：新集群的默认访问是 [Compute Engine 默认 ServiceAccount](https://cloud.google.com/compute/docs/access/service-accounts?hl=en_US#default_service_account)。 +- **允许完全访问所有 Cloud API**：通常，你只需设置云平台访问范围来允许完全访问所有 Cloud API，然后仅授予 ServiceAccount 相关的 IAM 角色。授予虚拟机实例的访问范围和授予 ServiceAccount 的 IAM 角色的组合决定了 ServiceAccount 对该实例的访问量。 +- **为每个 API 设置访问权限**：或者，你可以设置服务将调用的特定 API 方法的访问范围。 + +有关详细信息，请参阅[为 VM 启用 ServiceAccount](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances)。 + + +### 配置刷新间隔 + +刷新间隔可以通过 “gke-refresh” 来配置，它是一个代表秒的整数。 + +默认值为 300 秒。 + +你可以通过运行 `kubectl edit setting gke-refresh` 来更改同步间隔。 + +刷新窗口越短，争用条件发生的可能性就越小。但这确实增加了遇到 GCP API 可能存在的请求限制的可能性。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/rancher-server-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/rancher-server-configuration.md new file mode 100644 index 00000000000..bde06845d60 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/rancher-server-configuration.md @@ -0,0 +1,16 @@ +--- +title: Rancher Server Configuration +--- + + + + + +- [RKE1 集群配置](rke1-cluster-configuration.md) +- [RKE2 集群配置](rke2-cluster-configuration.md) +- [K3s 集群配置](k3s-cluster-configuration.md) +- [EKS 集群配置](eks-cluster-configuration.md) +- [AKS 集群配置](aks-cluster-configuration.md) +- [GKE 集群配置](gke-cluster-configuration/gke-cluster-configuration.md) +- [使用现有节点](use-existing-nodes/use-existing-nodes.md) +- [同步集群](sync-clusters.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md new file mode 100644 index 00000000000..c4e1efd5335 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md @@ -0,0 +1,141 @@ +--- +title: 在现有自定义节点上启动 Kubernetes +description: 要创建具有自定义节点的集群，你需要访问集群中的服务器，并根据 Rancher 的要求配置服务器。 +--- + + + + + +创建自定义集群时，Rancher 使用 RKE（Rancher Kubernetes Engine）在本地裸机服务器、本地虚拟机或云服务器节点中创建 Kubernetes 集群。 + +要使用此选项，你需要访问要在 Kubernetes 集群中使用的服务器。请根据[要求](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)配置每台服务器，其中包括硬件要求和 Docker 要求。在每台服务器上安装 Docker 后，你还需要在每台服务器上运行 Rancher UI 中提供的命令，从而将每台服务器转换为 Kubernetes 节点。 + +本节介绍如何设置自定义集群。 + +## 使用自定义节点创建集群 + +:::note 使用 Windows 主机作为 Kubernetes Worker 节点？ + +在开始之前，请参阅[配置 Windows 自定义集群](use-windows-clusters.md)。 + +::: + +### 1. 配置 Linux 主机 + +你可以通过配置 Linux 主机，来创建自定义集群。你的主机可以是： + +- 云虚拟机 +- 本地虚拟机 +- 裸机服务器 + +如果要重复使用之前的自定义集群中的节点，请在复用之前[清理节点](../../../../how-to-guides/new-user-guides/manage-clusters/clean-cluster-nodes.md)。如果你重复使用尚未清理的节点，则集群配置可能会失败。 + +根据[安装要求](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)和[生产就绪集群的检查清单](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md)配置主机。 + +如果你使用 Amazon EC2 作为主机，并希望使用[双栈 (dual-stack)](https://kubernetes.io/docs/concepts/services-networking/dual-stack/) 功能，则需要满足配置主机的其他[要求](https://rancher.com/docs/rke//latest/en/config-options/dual-stack#requirements)。 + +### 2. 创建自定义集群 + +1. 点击 **☰ > 集群管理**。 +1. 在**集群**页面上，单击**创建**。 +1. 单击**自定义**。 +1. 输入**集群名称**。 +1. 在**集群配置**中，选择 Kubernetes 版本、要使用的网络提供商，以及是否启用项目网络隔离。要查看更多集群选项，请单击**显示高级选项**。 + + :::note 你使用 Windows 主机作为 Kubernetes Worker 节点？ + + - 请参阅[启用 Windows 支持选项](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)。 + - 支持 Windows 集群的唯一网络插件是 Flannel。 + + ::: + + :::note Amazon EC2 上的双栈： + + 如果你使用 Amazon EC2 作为主机，并希望使用[双栈 (dual-stack)](https://kubernetes.io/docs/concepts/services-networking/dual-stack/) 功能，则需要满足配置 RKE 的其他[要求](https://rancher.com/docs/rke//latest/en/config-options/dual-stack#requirements)。 + + ::: + +6. 点击**下一步**。 + +4. 使用**成员角色**为集群配置用户授权。点击**添加成员**添加可以访问集群的用户。使用**角色**下拉菜单为每个用户设置权限。 + +7. 从**节点角色**中，选择要由集群节点充当的角色。你必须为 `etcd`、`worker` 和 `controlplane` 角色配置至少一个节点。自定义集群需要所有三个角色才能完成配置。有关角色的详细信息，请参阅[本节](../../../kubernetes-concepts.md#kubernetes-集群中节点的角色)。 + +:::note + +- 使用 Windows 主机作为 Kubernetes Worker 节点？请参阅[本节](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)。 +- 裸机服务器提醒：如果你想将裸机服务器专用于每个角色，则必须为每个角色配置一个裸机服务器（即配置多个裸机服务器）。 + +::: + +8. **可选**：点击[显示高级选项](../../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md)来指定注册节点时使用的 IP 地址，覆盖节点的主机名，或将[标签](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)或[污点](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)添加到节点。 + +9. 将屏幕上显示的命令复制到剪贴板。 + +10. 使用你惯用的 shell（例如 PuTTy 或远程终端）登录到你的 Linux 主机。粘贴剪贴板的命令并运行。 + +:::note + +如果要将特定主机专用于特定节点角色，请重复步骤 7-10。根据需要多次重复这些步骤。 + +::: + +11. 在 Linux 主机上运行完命令后，单击**完成**。 + +**结果**： + +你已创建集群，集群的状态是**配置中**。Rancher 已在你的集群中。 + +当集群状态变为 **Active** 后，你可访问集群。 + +**Active** 状态的集群会分配到两个项目： + +- `Default`：包含 `default` 命名空间 +- `System`：包含 `cattle-system`，`ingress-nginx`，`kube-public` 和 `kube-system` 命名空间。 + + +### 3. 仅限亚马逊：标签资源 + +如果你已将集群配置为使用 Amazon 作为**云提供商**，请使用集群 ID 标记你的 AWS 资源。 + +[Amazon 文档：标记你的 Amazon EC2 资源](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) + +:::note + +你可以使用 Amazon EC2 实例，而无需在 Kubernetes 中配置云提供商。如果你想使用特定的 Kubernetes 云提供商功能，配置云提供商即可。如需更多信息，请参阅 [Kubernetes 云提供商](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/)。 + +::: + +以下资源需要使用 `ClusterID` 进行标记： + +- **Nodes**：Rancher 中添加的所有主机。 +- **Subnet**：集群使用的子网。 +- **Security Group**：用于你的集群的安全组。 + +:::note + +不要标记多个安全组。创建 Elastic Load Balancer 时，标记多个组会导致错误。 + +::: + +应该使用的标签是： + +``` +Key=kubernetes.io/cluster/, Value=owned +``` + +`` 可以是你选择的任何字符串。但是，必须在你标记的每个资源上使用相同的字符串。将值设置为 `owned` 会通知集群所有带有 `` 标记的资源都由该集群拥有和管理。 + +如果你在集群之间共享资源，你可以将标签更改为： + +``` +Key=kubernetes.io/cluster/CLUSTERID, Value=shared +``` + +## 可选的后续步骤 + +创建集群后，你可以通过 Rancher UI 访问集群。最佳实践建议你设置以下访问集群的备用方式： + +- **通过 kubectl CLI 访问你的集群**：按照[这些步骤](../../../../how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#accessing-clusters-with-kubectl-from-your-workstation)在你的工作站上使用 kubectl 访问集群。在这种情况下，你将通过 Rancher Server 的认证代理进行认证，然后 Rancher 会让你连接到下游集群。此方法允许你在没有 Rancher UI 的情况下管理集群。 +- **通过 kubectl CLI 使用授权的集群端点访问你的集群**：按照[这些步骤](../../../../how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#authenticating-directly-with-a-downstream-cluster)直接使用 kubectl 访问集群，而无需通过 Rancher 进行认证。我们建议设置此替代方法来访问集群，以便在无法连接到 Rancher 时访问集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md new file mode 100644 index 00000000000..442efff377e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md @@ -0,0 +1,21 @@ +--- +title: 架构 +--- + + + + + +本章节重点介绍 [Rancher Server 及其组件](rancher-server-and-components.md) 以及 [Rancher 如何与下游 Kubernetes 集群通信](communicating-with-downstream-user-clusters.md)。 + +有关安装 Rancher 的不同方式的信息，请参见[安装选项概述](../../getting-started/installation-and-upgrade/installation-and-upgrade.md#安装方式概述)。 + +有关 Rancher API Server 的主要功能，请参见[概述](../../getting-started/overview.md#rancher-api-server-的功能)。 + +有关如何为 Rancher Server 设置底层基础架构，请参见[架构推荐](architecture-recommendations.md)。 + +:::note + +本节默认你已对 Docker 和 Kubernetes 有一定的了解。如果你需要了解 Kubernetes 组件如何协作，请参见 [Kubernetes 概念](../kubernetes-concepts.md)。 + +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/single-node-rancher-in-docker/single-node-rancher-in-docker.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/single-node-rancher-in-docker/single-node-rancher-in-docker.md new file mode 100644 index 00000000000..e272daa1ca4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/single-node-rancher-in-docker/single-node-rancher-in-docker.md @@ -0,0 +1,9 @@ +--- +title: Docker 中的单节点 Rancher +--- + + + + + +以下文档将讨论 Docker 安装的 [HTTP 代理配置](http-proxy-configuration.md)和[高级选项](advanced-options.md)。 From b14d053624d35ae3c2191e2fddece2cd483273a0 Mon Sep 17 00:00:00 2001 From: dingluyy <54788198+dingluyy@users.noreply.github.com> Date: Thu, 11 Apr 2024 04:52:25 +0800 Subject: [PATCH 43/67] Update missing Chinese translations of v2.8 and latest versions (#1221) * Update missing latest Chinese translations * Update missing v2.8 Chinese translations --- .../current.json | 12 + .../aws-cloud-marketplace.md | 36 + .../cloud-marketplace/cloud-marketplace.md | 11 + .../elemental/elemental.md | 30 + .../fleet/architecture.md | 11 + .../integrations-in-rancher/fleet/fleet.md | 27 + .../integrations-in-rancher/fleet/overview.md | 67 ++ .../fleet/use-fleet-behind-a-proxy.md | 58 ++ .../fleet/windows-support.md | 25 + .../harvester/harvester.md | 15 + .../harvester/overview.md | 42 + .../integrations-in-rancher.mdx | 51 ++ .../kubernetes-distributions.md | 34 + .../kubewarden/kubewarden.md | 31 + .../longhorn/longhorn.md | 15 + .../longhorn/overview.md | 74 ++ .../neuvector/neuvector.md | 27 + .../neuvector/overview.md | 196 +++++ .../rancher-desktop.md | 33 + .../version-2.8.json | 789 +++++++++--------- .../aws-cloud-marketplace.md | 36 + .../cloud-marketplace/cloud-marketplace.md | 11 + .../elemental/elemental.md | 30 + .../fleet/architecture.md | 11 + .../integrations-in-rancher/fleet/fleet.md | 27 + .../integrations-in-rancher/fleet/overview.md | 66 ++ .../fleet/use-fleet-behind-a-proxy.md | 58 ++ .../fleet/windows-support.md | 25 + .../harvester/harvester.md | 15 + .../harvester/overview.md | 42 + .../integrations-in-rancher.mdx | 51 ++ .../kubernetes-distributions.md | 34 + .../kubewarden/kubewarden.md | 31 + .../longhorn/longhorn.md | 15 + .../longhorn/overview.md | 74 ++ .../neuvector/neuvector.md | 27 + .../neuvector/overview.md | 196 +++++ .../rancher-desktop.md | 33 + 38 files changed, 1977 insertions(+), 389 deletions(-) create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/aws-cloud-marketplace.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cloud-marketplace/cloud-marketplace.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/elemental/elemental.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/architecture.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/fleet.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/windows-support.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/harvester/harvester.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/harvester/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/integrations-in-rancher.mdx create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/kubernetes-distributions/kubernetes-distributions.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/kubewarden/kubewarden.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/longhorn/longhorn.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/longhorn/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/neuvector/neuvector.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/neuvector/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/rancher-desktop.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/aws-cloud-marketplace.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cloud-marketplace/cloud-marketplace.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/elemental/elemental.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/architecture.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/fleet.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/windows-support.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/harvester/harvester.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/harvester/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/integrations-in-rancher.mdx create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/kubernetes-distributions/kubernetes-distributions.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/kubewarden/kubewarden.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/longhorn/longhorn.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/longhorn/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/neuvector/neuvector.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/neuvector/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/rancher-desktop.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current.json b/i18n/zh/docusaurus-plugin-content-docs/current.json index a1e3cb35335..4ce0df781ac 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current.json +++ b/i18n/zh/docusaurus-plugin-content-docs/current.json @@ -386,5 +386,17 @@ "sidebar.tutorialSidebar.category.Example Workflows": { "message": "工作流程示例", "description": "The label for category Example Workflows in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Virtualization on Kubernetes with Harvester": { + "message": "使用 Harvester 在 Kubernetes 上进行虚拟化", + "description": "The label for category Virtualization on Kubernetes with Harvester in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Cloud Native Storage with Longhorn": { + "message": "使用 Longhorn 进行云原生存储", + "description": "The label for category Cloud Native Storage with Longhorn in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Container Security with Neuvector": { + "message": "使用 NeuVector 实现容器安全", + "description": "The label for category Container Security with Neuvector in sidebar tutorialSidebar" } } diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/aws-cloud-marketplace.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/aws-cloud-marketplace.md new file mode 100644 index 00000000000..c7dd9264ee4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/aws-cloud-marketplace.md @@ -0,0 +1,36 @@ +--- +title: AWS Marketplace 集成 +--- + + + + + +## 概述 + +Rancher 提供与 AWS Marketplace 的集成，允许用户购买 SUSE 的支持合同。当你开始支持更多集群时，这种集成使你可以轻松调整支持需求。 + +## 限制 + +- 你必须运行 Rancher v2.6.7 或更高版本。 +- Rancher 的部署必须启用额外的指标。 +- Rancher 必须安装在 EKS 集群上。 +- 你必须通过 AWS Marketplace 购买至少一项 Rancher 支持的权限。 +- 你可能需要额外的设置来支持代理/离线用例。有关更多信息，请参阅[先决条件](adapter-requirements.md)。 + +## 如何使用 + +1. 完成[先决条件](adapter-requirements.md)。 +2. [安装 CSP Adapter](install-adapter.md)。 + +## 常见问题 + +**我以后可以购买对更多节点的支持吗？** + +可以。只需转到你最初用于购买支持的 AWS Marketplace 条目并增加权限数量即可。 + +**我可以在同一个 AWS 账户中使用多个 Rancher 实例吗？** + +可以。但是，Rancher 安装的每个集群都需要遵守先决条件。 + +此外，一个给定的权限一次只能由一个 Rancher 管理服务器使用。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cloud-marketplace/cloud-marketplace.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cloud-marketplace/cloud-marketplace.md new file mode 100644 index 00000000000..44c95be8fe9 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cloud-marketplace/cloud-marketplace.md @@ -0,0 +1,11 @@ +--- +title: 云市场集成 +--- + + + + + +Rancher 提供与云市场的集成，以便轻松购买对托管在某些云提供商上的安装支持。此外，此集成还提供了生成 Supportconfig Bundle 的功能，该 Bundle 可以给 rancher 提供支持。 + +此集成仅支持 AWS。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/elemental/elemental.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/elemental/elemental.md new file mode 100644 index 00000000000..6352de02ba1 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/elemental/elemental.md @@ -0,0 +1,30 @@ +--- +title: 使用 Elemental 进行操作系统管理 +--- + + + + + +Elemental 支持云原生主机管理。Elemental 允许你在任何位置装载任何机器，无论是在数据中心还是在边缘，并将它们无缝集成到 Kubernetes 中，同时管理你的工作流程（例如操作系统更新）。 + +## Elemental 和 Rancher + +Rancher 中的 Elemental： + +- 是 Kubernetes 原生的，它允许你通过 Kubernetes 集群中的 Elemental 管理操作系统。 +- 从 Kubernetes 的操作角度来看，不会造成干扰。 +- 是声明性的，并且 GitOps 友好。 +- 允许可信的、确定性的和可预测的 OCI Image-based flows。 +- 大规模运作。它支持 Fleet 规模的操作系统管理。 + +### 我什么时候应该使用 Elemental？ + +- Elemental 通过 Rancher manager 实现云原生操作系统管理。它适用于任何操作系统（例如，SLE Micro vanilla）。 +- Elemental 允许对数据中心和边缘的机器进行云原生管理。 +- Elemental 非常灵活，允许平台团队在其机器群中执行各种工作流。 + +## Elemental 和 Rancher Prime + +- 已作为 GUI 扩展深度集成到 Rancher 中。 +- 将 Rancher 的用例扩展到操作系统，如今 SLE Micro 与 Rancher 可以完美配合。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/architecture.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/architecture.md new file mode 100644 index 00000000000..ffdbff3edc2 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/architecture.md @@ -0,0 +1,11 @@ +--- +title: 架构 +--- + + + + + +Fleet 可以管理来自 Git 的原始 Kubernetes YAML、Helm Chart、Kustomize 或三者的任何组合的部署。无论来源如何，所有资源都会动态转化为 Helm Chart，并使用 Helm 作为引擎来部署到集群中的所有内容。这为你提供了高度的控制力、一致性和可审计性。Fleet 不仅关注扩展能力，而且还提供高度的控制和可见性，从而让用户准确了解集群上安装的内容。 + +![架构](/img/fleet-architecture.svg) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/fleet.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/fleet.md new file mode 100644 index 00000000000..8b1249e41a0 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/fleet.md @@ -0,0 +1,27 @@ +--- +title: 使用 Fleet 进行持续交付 +--- + + + + + +Fleet 通过集群 Fleet 的供应链协调和管理应用程序的持续交付。Fleet 使用 GitOps 作为一种安全的运营模式，组织供应链，帮助团队及时自信地交付。 + +## Fleet 和 Rancher + +许多用户经常同时管理 10 个以上的集群。鉴于集群的激增，持续交付是 Rancher 的重要组成部分。Fleet 使用 GitOps 确保可靠的持续交付体验，这是一种安全且越来越常见的运营模式。 + +### 我什么时候应该使用 Fleet? + +- 我需要跨地理区域部署我的监控堆栈（例如 Grafana、Prometheus），每个区域都有不同的保留策略。 +- 我是一名平台运营商，希望使用可扩展且安全的操作模型（GitOps）为集群配置所有组件。 +- 我是一名应用程序开发人员，希望我的最新更改自动进入我的开发环境。 + +## Fleet 和 Rancher Prime + +Fleet 已经作为持续交付工具和 GitOps 引擎深度集成到 Rancher 中。 + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/overview.md new file mode 100644 index 00000000000..cedc5ee7a36 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/overview.md @@ -0,0 +1,67 @@ +--- +title: 概述 +--- + + + + + +使用 Fleet 进行持续交付是大规模的 GitOps。Fleet 旨在管理多达一百万个集群。它也足够轻量级，对于[单个集群](https://fleet.rancher.io/installation#default-install)也很有效，但当你达到[大规模](https://fleet.rancher.io/installation#configuration-for-multi-cluster)时，它真的会大放异彩。大规模是指单个组织中的大量集群、大量部署或大量团队。 + +Fleet 是 Rancher 的一个独立项目，可以通过 Helm 安装在任何 Kubernetes 集群上。 + +## 架构 + +有关 Fleet 如何运作的信息，请参阅[架构](./architecture)页面。 + +## 在 Rancher UI 中访问 Fleet + +Fleet 预安装在 Rancher 中，并由 Rancher UI 中的**持续交付**选项进行管理。有关持续交付和其他 Fleet 故障排除技巧的更多信息，请参阅[此处](https://fleet.rancher.io/troubleshooting)。 + +用户可以按照 **gitops** 实践，利用持续交付将应用程序部署到 git 仓库中的 Kubernetes 集群，而无需任何手动操作。 + +按照以下步骤在 Rancher UI 中访问持续交付： + +1. 单击 **☰ > 持续交付**。 + +1. 在菜单顶部选择命名空间 ，注意以下事项： + + - 默认情况下，选择 **fleet-default**，包括通过 Rancher 注册的所有下游集群。 + + - 你可以切换到 **fleet-local**，它仅包含 **local** 集群，或者你可以创建自己的工作空间，将集群分配和移动到其中。 + + - 然后，你可以通过单击左侧导航栏上的**集群**来管理集群。 + +1. 单击左侧导航栏上的 **Gitrepos**，将 gitrepo 部署到当前工作空间的集群中。 + +1. 选择 [git 仓库](https://fleet.rancher.io/gitrepo-add)和[目标集群/集群组](https://fleet.rancher.io/gitrepo-targets)。你也可以通过单击左侧导航栏中的**集群组**在 UI 中创建集群组。 + +1. 部署 gitrepo 后，你可以通过 Rancher UI 监控应用程序。 + +## Windows 支持 + +有关对具有 Windows 节点的集群的支持的详细信息，请参阅 [Windows 支持](./windows-support)页面。 + +## GitHub 仓库 + +Fleet Helm charts 可在[此处](https://github.com/rancher/fleet/releases)获取。 + +## 在代理后使用 Fleet + +有关在代理后面使用 Fleet 的详细信息，请参阅[在代理后使用 Fleet](./use-fleet-behind-a-proxy) 页面。 + +## Helm Chart 依赖 + +为了成功部署具有依赖项的 Helm Chart，你必须运行一个手动命令（如下所示），因为用户需要满足依赖列表。如果不执行此操作，继续克隆你的仓库并运行 `helm install`，则会因依赖项丢失而导致安装失败。 + +git 仓库中的 Helm Chart 必须在 Chart 子目录中包含其依赖。 你必须手动运行 `helm dependencies update $chart` 或在本地运行 `helm dependencies build $chart`，然后将完整的 Chart 目录提交到 git 仓库。请注意，你需要使用适当的参数更新你的命令。 + +## 故障排除 + +- **已知问题**：Fleet gitrepos 的 clientSecretName 和 helmSecretName 密文不包含在 [backup-restore-operator](../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md#1-install-the-rancher-backup-operator) 创建的备份或恢复中。一旦有永久的解决方案，我们将更新社区内容。 + +- **临时解决方法**：默认情况下，用户定义的密文不会在 Fleet 中备份。如果执行灾难恢复或将 Rancher 迁移到新集群，则有必要重新创建密文。要修改 ResourceSet 以包含要备份的额外资源，请参阅文档[此处](https://github.com/rancher/backup-restore-operator#user-flow)。 + +## 文档 + +Fleet 文档位于 https://fleet.rancher.io/ 。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md new file mode 100644 index 00000000000..3806cbb2248 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md @@ -0,0 +1,58 @@ +--- +title: 在代理后使用 Fleet +--- + + + + + +在本节中，你将学习如何在具有公共 IP 的 Rancher 服务器，以及一个没有公共 IP 但配置为使用代理的 Kubernetes 集群的设置中启用 Fleet。 + +Rancher 不会与已注册的下游集群建立连接。部署在下游集群上的 Rancher agent 必须能够与 Rancher 建立连接。 + +要让 Fleet 在代理后工作，你需要为下游集群设置 **Agent 环境变量**。以下是集群级别的配置选项。 + +你可以通过 Rancher UI 为任何集群类型（包括注册集群和自定义集群）配置这些环境变量。可以在编辑现有集群或配置新集群时添加变量。 + +对于公共下游集群，[在 Rancher UI 中设置必要的环境变量](#在-rancher-ui-中设置环境变量)就足够了。 + +对于私有节点或私有集群，则需要在节点上设置环境变量。然后，在配置自定义集群或注册私有集群时，在 Rancher UI 中配置环境变量。有关如何在 K3s 集群中的 Ubuntu 节点上设置环境变量的示例，请参阅[本节](#在私有节点上设置环境变量)。 + +## 必要的环境变量 + +为代理添加 Fleet agent 环境变量时，将 替换为你的私有代理 IP。 + +| 变量名称 | 值 | +| ------------- | ----------------------------------------------------------------------- | +| `HTTP_PROXY` | http://:8888 | +| `HTTPS_PROXY` | http://:8888 | +| `NO_PROXY` | 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local | + +## 在 Rancher UI 中设置环境变量 + +要将环境变量添加到现有集群： + +1. 单击 **☰ > 集群管理**。 +1. 转到要添加环境变量的集群，然后单击 **⋮ > 编辑配置**。 +1. 单击**高级选项**。 +1. 单击**添加环境变量**。 +1. 输入[必要的环境变量](#必要的环境变量) +1. 单击**保存**。 + +**结果：** Fleet agent 会在代理后工作。 + +## 在私有节点上设置环境变量 + +对于私有节点和私有集群，需要在节点本身上设置代理环境变量，并在 Rancher UI 中进行配置。 + +此示例显示了如何在 K3s 集群中的 Ubuntu 节点上设置环境变量： + +``` +ssh -o ForwardAgent=yes ubuntu@ +ssh +export proxy_private_ip= +export HTTP_PROXY=http://${proxy_private_ip}:8888 +export HTTPS_PROXY=http://${proxy_private_ip}:8888 +export NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/windows-support.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/windows-support.md new file mode 100644 index 00000000000..7102f0fa01d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/fleet/windows-support.md @@ -0,0 +1,25 @@ +--- +title: Windows 支持 +--- + + + + + +在 Rancher v2.5.6 之前，`agent` 在具有 Windows 节点的下游集群上没有原生的 Windows 清单。这将导致集群的 `agent` pod 运行失败。 + +如果你从旧版本的 Rancher 升级到 v2.5.6+，你可以在 _下游集群_ 中部署具有以下工作流的可运行的 `agent`： + +1. 封锁所有 Windows 节点。 +1. 对 `agent` 工作负载应用以下容忍度。 +1. 取消所有 Windows 节点的封锁。 +1. 删除所有 `agent` pod。使用新的容忍度来创建新 pod。 +1. 一旦 `agent` pod 运行，并且 Fleet 的自动更新已启用，它们会更新到兼容 Windows 的 `agent` 版本。 + +```yaml +tolerations: + - effect: NoSchedule + key: cattle.io/os + operator: Equal + value: linux +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/harvester/harvester.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/harvester/harvester.md new file mode 100644 index 00000000000..8f4b2312b56 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/harvester/harvester.md @@ -0,0 +1,15 @@ +--- +title: 使用 Harvester 在 Kubernetes 上进行虚拟化 +--- + + + + + +## Harvester + +Harvester 是 Rancher v2.6.1 新增的功能，是基于 Kubernetes 构建的开源超融合基础架构（HCI）软件。Harvester 安装在裸金属服务器上，提供集成的虚拟化和分布式存储功能。虽然 Harvester 使用 Kubernetes 运行，但它不需要用户了解 Kubernetes 概念，这使得它更加用户友好。 + +## Harvester 与 Rancher + +凭借 Rancher Prime 和 Harvester，IT 运维人员现在可以访问一个企业级的、易于使用的基础设施平台，该平台可以协同管理他们的虚拟机和 Kubernetes 集群。有关产品支持的更多信息，请参阅[支持矩阵](https://www.suse.com/suse-harvester/support-matrix/all-supported-versions/harvester-v1-2-0/)。通过 Rancher 虚拟化管理功能，用户可以导入和管理多个 Harvester 集群。利用 Rancher 的认证功能和 RBAC 控制来支持多租户。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/harvester/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/harvester/overview.md new file mode 100644 index 00000000000..ac720bec060 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/harvester/overview.md @@ -0,0 +1,42 @@ +--- +title: 概述 +--- + + + + + +[Harvester](https://docs.harvesterhci.io/) 是 Rancher v2.6.1 新增的功能，是基于 Kubernetes 构建的开源超融合基础架构（HCI）软件。Harvester 安装在裸金属服务器上，提供集成的虚拟化和分布式存储功能。虽然 Harvester 使用 Kubernetes 运行，但它不需要用户了解 Kubernetes 概念，这使得它更加用户友好。 + +### 功能开关 + +Harvester 功能开关用于管理对 Rancher 中虚拟化管理（VM）页面的访问，用户可以直接导航到 Harvester 集群并访问 Harvester UI。Harvester 的功能开关默认启用。如需了解 Rancher 中功能开关的更多详细信息，请单击[此处](../../how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 + +要导航到 Harvester 集群，请单击 **☰ > 虚拟化管理**。在 Harvester 集群页面中，单击集群以转到该 Harvester 集群的视图。 + +- 如果启用了 Harvester 功能开关，Harvester 集群将会从列出 Kubernetes 集群的任何页面或应用（例如 Fleet 和多集群应用）中过滤掉。 + +- 如果禁用了 Harvester 功能开关，并且导入了 Harvester 集群，Harvester 集群将显示在集群管理页面的 Rancher 集群列表中。仅当功能开关为关闭时，Harvester 集群才会显示在集群列表中。 + +- 集成 Harvester 后，你可以将 Harvester 集群导入 Rancher，对应的集群类型是 `Harvester`。 + +- 用户只能在虚拟化管理页面上导入 Harvester 集群。不支持在集群管理页面上导入集群，并且会出现警告，建议你返回虚拟化管理页面执行此操作。 + +### Harvester 主机驱动 + +[Harvester 主机驱动](https://docs.harvesterhci.io/v1.1/rancher/node/node-driver/)通常可用于 Rancher 中的 RKE 和 RKE2 选项。无论 Harvester 功能开关是否启用，主机驱动都是可用的。请注意，主机驱动默认处于关闭状态。用户只能通过集群管理页面在 Harvester 上创建 RKE 或 RKE2 集群。 + +Harvester 允许通过 Harvester UI 上传和显示 `.ISO` 镜像，但 Rancher UI 是不支持的。这是因为 `.ISO` 镜像通常需要额外的设置，这会干扰干净的部署（即无需用户干预），并且它们通常不用于云环境。 + +如需了解 Rancher 中主机驱动的更多详细信息，请单击[此处](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/about-provisioning-drivers#主机驱动)。 + +### 端口要求 + +Harvester 集群的端口要求可以在[此处](https://docs.harvesterhci.io/v1.1/install/requirements#networking)找到。 + +此外，其他网络注意事项如下： + +- 请务必为 VM VLAN 网络启用物理交换机的 VLAN 中继端口。 +- 按照[此处](https://docs.harvesterhci.io/v1.1/networking/index)的网络设置指南进行操作。 + +对于其他集群（例如 K3s 和 RKE1）的其他端口要求，请参阅[这些文档](https://docs.harvesterhci.io/v1.1/install/requirements/#guest-clusters)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/integrations-in-rancher.mdx b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/integrations-in-rancher.mdx new file mode 100644 index 00000000000..c0a824cc9da --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/integrations-in-rancher.mdx @@ -0,0 +1,51 @@ +--- +title: Rancher 中的集成 +--- + + + + + +import { Card, CardSection } from "@site/src/components/CardComponents"; +import { RocketRegular } from "@fluentui/react-icons"; + +Prime 是 Rancher 生态系统的企业级产品，具有更高的安全性、更长的生命周期和对 Prime 专有文档的访问权限。Rancher Prime 安装资产托管在受信任的 SUSE 注册表上，由 Rancher 拥有和管理。受信任的 Prime 注册表仅包括经过社区测试的稳定版本。 + +Prime 还提供生产支持选项，以及根据你的商业需求定制的订阅附加组件。 + +要了解更多信息并开始使用 Rancher Prime，请访问[本页](https://www.rancher.com/quick-start)。 + +}> + + + + + + + + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/kubernetes-distributions/kubernetes-distributions.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/kubernetes-distributions/kubernetes-distributions.md new file mode 100644 index 00000000000..b45cce6b4d7 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/kubernetes-distributions/kubernetes-distributions.md @@ -0,0 +1,34 @@ +--- +title: Kubernetes 发行版 +--- + + + + + +## K3s + +K3s 是一款轻量级、完全兼容的 Kubernetes 发行版，专为一系列用例设计，包括边缘计算、物联网、CI/CD、开发和将 Kubernetes 嵌入到应用程序中。它将系统打包为单个二进制文件，使用 sqlite3 作为默认存储，并提供用户友好的启动器，从而简化了 Kubernetes 的管理。K3s 包括 Local Storage 和负载均衡、Helm Chart 控制器和 Traefik CNI 等基本功能。它最大限度地减少了外部依赖，并提供了精简的 Kubernetes 体验。K3s 于 2020 年 6 月作为沙箱项目捐赠给 CNCF。 + +### K3s 与 Rancher + +- Rancher 允许在一系列平台上轻松配置 K3s，包括 Amazon EC2、DigitalOcean、Azure、vSphere 或现有服务器。 +- Kubernetes 集群的标准 Rancher 管理，包括所有概述[集群管理功能](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup#cluster-management-capabilities-by-cluster-type)。 + +## RKE2 + +RKE2 是 Rancher 开发的一款兼容 Kubernetes 的发行版。它是专门为美国联邦政府部门的安全性和合规性而设计的。 + +RKE2 的主要特性包括: + +1. **安全性和合规性重点**：RKE2 非常重视安全性和合规性，在“默认安全”框架下运行，适用于政府服务以及金融和医疗保健等高度监管的行业。 +1. **CIS Kubernetes Benchmark 一致性**：RKE2 经过预配置，符合 CIS Kubernetes Hardening Benchmark（目前支持 v1.23 和 v1.7），并且只需最少的手动干预。 +1. **FIPS 140-2 合规性**：RKE2 符合 FIPS 140-2 标准，使用经过 FIPS 验证的加密模块作为其组件。 +1. **嵌入式 ETCD**：RKE2 默认使用嵌入式 ETCD 作为其数据存储。这使其与标准的 Kubernetes 实践更加紧密地结合在一起，从而可以更好地与其他 Kubernetes 工具集成并降低错误配置的风险。 +1. **与上游 Kubernetes 保持一致**：RKE2 旨在与上游 Kubernetes 保持密切一致，降低在使用偏离标准 Kubernetes 实践的发行版时可能出现的不一致风险。 +1. **多重 CNI 支持**：RKE2 支持多个容器网络接口（CNI）插件，包括 Cilium、Calico 和 Multus。这对于具有各种生产设施的电信分发中心和工厂等用例至关重要。 + +## RKE2 与 Rancher + +- Rancher 允许在一系列平台上轻松配置 RKE2，包括 Amazon EC2、DigitalOcean、Azure、vSphere 或现有服务器。 +- Kubernetes 集群的标准 Rancher 管理，包括所有概述[集群管理功能](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup#cluster-management-capabilities-by-cluster-type)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/kubewarden/kubewarden.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/kubewarden/kubewarden.md new file mode 100644 index 00000000000..80c6d16d315 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/kubewarden/kubewarden.md @@ -0,0 +1,31 @@ +--- +title: 使用 Kubewarden 进行高级策略管理 +--- + + + + + +Kubewarden 是一个策略引擎，用于保护和帮助管理集群资源。它允许通过策略验证和更改资源请求，包括上下文感知策略和验证镜像签名。它可以在监视或强制模式下运行策略，并提供集群状态的概述。 + +Kubewarden 旨在通过启用和简化“策略即代码”来成为通用策略引擎。Kubewarden 策略被编译到 WebAssembly 中：它们体积小（400KB~2MB），沙箱式，安全且便携。它旨在通过迎合组织中的每个人来实现通用性： + +- 策略用户：使用 Kubernetes 自定义资源管理和声明策略，重新使用 Rego（OPA 和 Gatekeeper）中编写的现有策略。在 CI/CD 中测试群集外部的策略。 +- 策略开发人员：使用你喜欢的 Wasm 编译语言（Rego、Go、Rust、C#、Swift、Typescript 等）编写策略。重新使用你已经熟悉的工具、库和工作流的生态系统。 +- 策略分发器：策略是 OCI 工件，通过 OCI 仓库为它们提供服务，并在你的基础设施中使用行业标准，如 Software-Bill-Of-Materials 和工件签名。 +- 集群操作员：Kubewarden 是模块化的（OCI 注册表、PolicyServers、Audit Scanner、Controller）。配置你的部署以满足你的需求，隔离不同的租户。使用审核扫描程序和策略报告在集群中获取过去、当前和可能的违规行为的概述。 +- Kubewarden Integrator：将其用作编写新的 Kubewarden 模块和自定义策略的平台。 + +## Kubewarden 和 Rancher + +Kubewarden 的上游 Helm Chart 完全集成为 Rancher 应用程序，为安装选项提供了 UI。这些 Chart 还附带了尊重 Rancher 堆栈的默认设置（例如：不监管 Rancher 系统命名空间）以及默认的 PolicyServer 和策略。用户可以访问所有 Kubewarden 功能，并可以通过与 Kubernetes API 交互（例如：使用 kubectl）手动部署 PolicyServer 和策略。 + +Kubewarden 提供了对已删除的 Kubernetes Pod 安全策略的完全替换。 Kubewarden 还通过增强其安全功能，与最新版本的 Kubernetes 引入的新 Pod Security Admission 功能进行了集成。 + +## Kubewarden 和 Rancher Prime + +Kubewarden 的 Rancher UI 扩展将其集成到 Rancher UI 中。UI 扩展可自动安装和配置 Kubewarden 堆栈，并配置对 SUSE 维护的策略的访问。UI 扩展提供了对现成策略的策划目录的访问。使用 UI 扩展，人们可以浏览、安装和配置这些策略。 + +UI 扩展提供了 Kubewarden 堆栈组件及其行为的概述。这包括对 Kubewarden 指标和跟踪事件的访问。操作员可以了解策略对集群的影响并排查问题。 + +此外，UI 扩展还提供了 Policy Reporter UI，它可以直观地概述 Kubernetes 集群的合规性状态。通过此 UI，操作员可以快速识别所有不合规的 Kubernetes 资源，了解违规原因并采取相应行动。所有这一切都得益于 Rancher Prime 的支持。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/longhorn/longhorn.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/longhorn/longhorn.md new file mode 100644 index 00000000000..c3727f30ff8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/longhorn/longhorn.md @@ -0,0 +1,15 @@ +--- +title: 使用 Longhorn 进行云原生存储 +--- + + + + + +## Longhorn + +Longhorn 是官方的[云原生计算基金会（CNCF）](https://cncf.io/)项目，它为 Kubernetes 提供了一个可以在任何地方运行的强大的云原生分布式存储平台。当与 Rancher 相结合使用时，Longhorn 使你可以在 Kubernetes 环境中轻松、快速且可靠地部署高可用的持久块存储。 + +## Longhorn 与 Rancher + +凭借 Rancher Prime 和 Longhorn，用户可以通过 Rancher 应用商店一键轻松部署，并对托管集群进行生命周期管理；允许用户能够安装和升级，同时进行清空操作以实现优雅的操作。Longhorn 和 Rancher Prime 还提供了 Windows 的混合集群支持，Rancher 托管的镜像，通过 Rancher 的 UI 代理访问，以及使用 Longhorn 指标进行 Rancher 监控。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/longhorn/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/longhorn/overview.md new file mode 100644 index 00000000000..b5298c3c21d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/longhorn/overview.md @@ -0,0 +1,74 @@ +--- +title: 概述 +--- + + + + + +[Longhorn](https://longhorn.io/) 是一个轻量级、可靠且易于使用的 Kubernetes 分布式块存储系统。 + +Longhorn 是免费的开源软件。它最初由 Rancher Labs 开发，现在被开发为云原生计算基金会的沙箱项目。它可以通过 Helm、kubectl 或 Rancher UI 安装在任何 Kubernetes 集群上。有关其架构的更多信息，请参阅[此处](https://longhorn.io/docs/latest/concepts/)。 + +使用 Longhorn，你可以： + +- 使用 Longhorn 卷作为 Kubernetes 集群中分布式有状态应用程序的持久存储 +- 将块存储分区为 Longhorn 卷，以便你可以在有或没有云提供商的情况下使用 Kubernetes 卷 +- 跨多个节点和数据中心复制块存储以提高可用性 +- 将备份数据存储在 NFS 或 AWS S3 等外部存储中 +- 创建跨集群灾难恢复卷，以便使用另一个 Kubernetes 集群中的备份快速恢复主 Kubernetes 集群中的数据 +- 计划卷的定期快照，并将定期备份调度到 NFS 或兼容 S3 的辅助存储 +- 使用备份来恢复卷 +- 在不中断持久卷的情况下升级 Longhorn + +
Longhorn 仪表板
+ +![Longhorn 仪表板](/img/longhorn-screenshot.png) + +### 使用 Rancher 安装 Longhorn + +1. 满足所有[安装要求](https://longhorn.io/docs/latest/deploy/install/#installation-requirements)。 +1. 转到要安装 Longhorn 的集群。 +1. 单击 **Apps**。 +1. 单击 **Charts**。 +1. 单击 **Longhorn**。 +1. 可选：要自定义初始设置，请单击 **Longhorn 默认设置**并编辑配置。如需自定义设置的帮助，请参阅 [Longhorn 文档](https://longhorn.io/docs/latest/references/settings/)。 +1. 单击**安装**。 + +**结果**：Longhorn 已部署到 Kubernetes 集群中。 + +### 从 Rancher UI 访问 Longhorn + +1. 转到安装了 Longhorn 的集群。在左侧导航菜单中，单击 **Longhorn**。 +1. 在此页面上，你可以编辑 Longhorn 管理的 Kubernetes 资源。要查看 Longhorn UI，请单击**概述**中的 **Longhorn** 按钮。 + +**结果**：你将转到 Longhorn UI，在这里你可以管理 Kubernetes 集群中的 Longhorn 卷及其副本，以及可能存在于另一个 Kubernetes 集群或 S3 中的 Longhorn 存储辅助备份。 + +### 从 Rancher UI 卸载 Longhorn + +1. 转到安装了 Longhorn 的集群，然后单击 **Apps**。 +1. 点击**已安装的应用**。 +1. 转到 `longhorn-system` 命名空间并选中 `longhorn` 和 `longhorn-crd` 应用程序旁边的框。 +1. 单击**删除**并确认**删除**。 + +**结果**：Longhorn 已被卸载。 + +### GitHub 仓库 + +Longhorn 项目可在[此处](https://github.com/longhorn/longhorn)获取。 + +### 文档 + +Longhorn 文档在[此处](https://longhorn.io/docs/)。 + +### 架构 + +Longhorn 为每个卷创建专用的存储控制器，并在多个节点上存储的多个副本之间同步复制该卷。 + +存储控制器和副本本身是使用 Kubernetes 编排的。 + +有关其架构的更多信息，请参阅[此处](https://longhorn.io/docs/latest/concepts/)。 + +
Longhorn 架构
+ +![Longhorn 架构](/img/longhorn-architecture.svg) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/neuvector/neuvector.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/neuvector/neuvector.md new file mode 100644 index 00000000000..d2f1257309f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/neuvector/neuvector.md @@ -0,0 +1,27 @@ +--- +title: 使用 NeuVector 实现容器安全 +--- + + + + + +NeuVector 是唯一一个100%开源、零信任的容器安全平台。在整个容器生命周期中持续扫描。清除安全路障。从一开始就制定安全策略，以最大限度地提高开发人员的灵活性。NeuVector 提供从构建到生产的漏洞和合规性扫描与管理。独特的 NeuVector 运行时保护通过七层容器防火墙保护集群内的网络连接以及集群的入口/出口。此外，NeuVector 还监视容器和主机上的进程和文件活动，以阻止未经授权的活动。 + +## NeuVector 和 Rancher + +所有 NeuVector 功能均可通过 Rancher 进行集成部署并单点登录 NeuVector 控制台。Rancher 集群管理员能够在其集群上部署和管理 NeuVector 部署，Helm 值、configMaps、自定义资源定义（CRD）和 NeuVector 控制台轻松配置 NeuVector。 + +使用 NeuVector 和 Rancher： + +- 部署、管理和保护多个集群。 +- 管理和报告 Rancher 工作负载和节点的漏洞和合规性结果。 + +## NeuVector Prime 和 Rancher Prime + +Rancher Manager 的 NeuVector UI 扩展可用于并支持 Rancher Prime 和 NeuVector Prime 客户。此扩展提供： + +- NeuVector 自动化部署，包括 Rancher Prime NeuVector 扩展仪表板。 +- 访问每个集群的重要安全信息，如关键安全事件、漏洞扫描结果和入口/出口暴露。 +- 直接对 Rancher 资源（例如节点和容器/ Pod）进行集成漏洞 (CVE) 和合规扫描。 +- 集成操作，如手动触发 Rancher 资源的扫描。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/neuvector/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/neuvector/overview.md new file mode 100644 index 00000000000..c6627aa62a0 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/neuvector/overview.md @@ -0,0 +1,196 @@ +--- +title: 概述 +--- + + + + + +### Rancher 中的 NeuVector 集成 + +[NeuVector 5.x](https://open-docs.neuvector.com/) 是一个开源的，以容器为中心的安全应用程序，Rancher 已集成 NeuVector。NeuVector 在运行时为关键应用程序和数据提供实时的合规、可见和保护功能。NeuVector 提供具有 CIS Benchmark 和漏洞扫描的防火墙、容器进程/文件系统监控和安全审计。有关 Rancher 安全性的更多信息，请参阅[安全文档](../../reference-guides/rancher-security)。 + +NeuVector 可以通过 Helm Chart 启用。你可以在 **Apps** 或 Rancher UI 中的 **Cluster Tools** 中安装该 Chart。安装 Helm Chart 后，用户可以轻松地[在 Rancher 中部署和管理 NeuVector 集群](https://open-docs.neuvector.com/deploying/rancher#deploy-and-manage-neuvector-through-rancher-apps-marketplace)。 + +### 使用 Rancher 安装 NeuVector + +Harvester Helm Chart 用于管理 Rancher 中 NeuVector UI 的访问，用户可以在 Rancher 中直接跳转，然后部署和管理 NeuVector 集群。 + +**通过 Apps 导航并安装 NeuVector Chart：** + +1. 单击 **☰ > 集群管理**。 +1. 在 Clusters 页面上，转到要部署 NeuVector 的集群，然后单击 **Explore**。 +1. 转到 **Apps > Charts**，然后从 Chart 仓库中安装 **NeuVector**。 +1. 不同的集群类型需要不同的容器运行时。配置 Helm Chart 值时，转到**容器运行时**，然后根据集群类型选择运行时。最后，再次单击**安装**。 + +以下是一些例子： + +- RKE1：`docker` +- K3s 和 RKE2：`k3scontainerd` +- AKS：`containerd` 适用于 v1.19 及更高版本 +- EKS：`docker` 适用于 v1.22 及以下版本；`containerd` 适用于 v1.23 及更高版本 +- GKE：`containerd` (请参阅 [Google 文档](https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd)了解更多信息) + + :::note + + 在安装过程中一次只能选择一个容器运行时引擎。 + + ::: + +**通过集群工具导航并安装 NeuVector Chart：** + +1. 单击 **☰ > 集群管理**。 +1. 在 Clusters 页面上，转到要部署 NeuVector 的集群，然后单击 **Explore**。 +1. 点击左侧导航栏底部的**集群工具**。 +1. 按照上面的步骤 4 相应地选择你的容器运行时，然后再次单击**安装**。 + +### 从 Rancher UI 访问 NeuVector + +1. 导航到安装了 NeuVector 的集群的 Cluster Explorer。在左侧导航栏中，单击 **NeuVector**。 +1. 单击外部链接以转到 NeuVector UI。选择链接后，用户必须接受`最终用户许可协议`才能访问 NeuVector UI。 + +### 从 Rancher UI 卸载 NeuVector + +**通过 Apps 卸载：** + +1. 单击 **☰ > 集群管理**。 +1. 在 **Apps** 下，点击 **Installed Apps**。 +1. 在 `cattle-neuvector-system` 下，选择 NeuVector 应用程序（如果需要，还可以选择相关的 CRD），然后单击**删除**。 + +**通过集群工具卸载：** + +1. 单击 **☰ > 集群管理**。 +1. 单击屏幕左下角的**集群工具**，然后单击 NeuVector Chart 下方的垃圾桶图标。如果需要，选择`删除与此应用关联的 CRD`，然后单击**删除**。 + +### GitHub 仓库 + +NeuVector 项目在[这里](https://github.com/neuvector/neuvector)。 + +### 文档 + +NeuVector 文档在[这里](https://open-docs.neuvector.com/)。 + +### 架构 + +NeuVector 安全解决方案包含四种类型的安全容器，分别是 Controller、Enforcer、Manager 和 Scanner。它还提供了一个称为 All-in-One 的特殊容器（主要用于 Docker 原生部署），能将 Controller、Enforcer 和 Manager 功能组合在一个容器中。此外，还有一个 Updater，运行该程序时会更新 CVE 数据库。 + +- **Controller：** 管理 NeuVector Enforcer 容器；为管理控制台提供 REST API。 +- **Enforcer：** 执行安全策略。 +- **Manager：** 提供一个 web-UI 和 CLI 控制台来管理 NeuVector 平台。 +- **All-in-One：** 包括 Controller、Enforcer 和 Manager。 +- **Scanner：** 对镜像、容器和节点执行漏洞和合规性扫描。 +- **Updater：** 更新 Neuvector 的 CVE 数据库（运行的时候）；重新部署 scanner pod。 + +
**NeuVector 安全容器：**
+ +![NeuVector 安全容器](/img/neuvector-security-containers.png) + +
**NeuVector 架构：**
+ +![NeuVector 架构](/img/neuvector-architecture.png) + +要了解有关 NeuVector 架构的更多信息，请参阅[此处](https://open-docs.neuvector.com/basics/overview#architecture)。 + +### CPU 和内存分配 + +以下是默认 NeuVector Chart 安装部署的最低计算资源推荐。请注意，未设置资源限制。 + +| 容器 | CPU - 请求 | 内存 - 请求 | +| ---------- | ----------------------------- | ----------- | +| Controller | 3（每个控制器需要 1GB 1vCPU） | \* | +| Enforcer | 所有节点上 (500MB .5vCPU) | 1GB | +| Manager | 1 (500MB .5vCPU) | \* | +| Scanner | 3 (100MB .5vCPU) | \* | + +\* Controller、Manager 和 Scanner 容器合计至少需要 1GB 内存。 + +### 强化集群支持 - Calico 和 Canal + + + + +- 如果 PSP 设置为 true，则所有 NeuVector 组件都是可部署的。 + +你需要为强化集群环境进行额外的配置，如下所示： + +1. 单击 **☰ > 集群管理**。 +1. 选择你创建的集群，并点击 **Explore**。 +1. 在左侧导航栏中，点击 **Apps**。 +1. 安装（或升级到）NeuVector 版本 `100.0.1+up2.2.2`。 + +- 在**编辑选项** > **其它配置**下，选中复选框来启用 **Pod 安全策略**（无需其他配置）： + + ![为 RKE1 强化集群启用 PSP](/img/psp-nv-rke.png) + +1. 点击右下角的**安装**。 + + + + +- 如果 PSP 设置为 true，则可以部署 NeuVector 组件 Controller 和 Enforcer。 + +**仅适用于 NeuVector Chart 版本 100.0.0+up2.2.0：** + +- 对于 Manager、Scanner 和 Updater 组件，需要进行额外的配置，如下所示： + +``` +kubectl patch deploy neuvector-manager-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}' +kubectl patch deploy neuvector-scanner-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}' +kubectl patch cronjob neuvector-updater-pod -n cattle-neuvector-system --patch '{"spec":{"jobTemplate":{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}}}' +``` + +
+ +你需要为强化集群环境进行额外的配置。 + +> **注意：**你必须更新 RKE2 和 K3s 强化集群中的配置，如下所示。 + +1. 单击 **☰ > 集群管理**。 +1. 选择你创建的集群，并点击 **Explore**。 +1. 在左侧导航栏中，点击 **Apps**。 +1. 安装（或升级到）NeuVector 版本 `100.0.1+up2.2.2`。 + +- 在**编辑选项** > **其它配置**下，选中复选框来启用 **Pod 安全策略**。请注意，对于 `Manager runAsUser ID`，`Scanner runAsUser ID` 和 `Updater runAsUser ID`，你还必须输入大于 `0` 的值： + + ![为 RKE2 和 K3s 强化集群启用 PSP](/img/psp-nv-rke2.png) + +1. 点击右下角的**安装**。 + + + + +### 启用 SELinux 的集群支持 - Calico 和 Canal + +要在 RKE2 集群上启用 SELinux，请执行以下步骤： + +- 如果 PSP 设置为 true，则可以部署 NeuVector 组件 Controller 和 Enforcer。 + +**仅适用于 NeuVector Chart 版本 100.0.0+up2.2.0：** + +- 对于 Manager、Scanner 和 Updater 组件，需要进行额外的配置，如下所示： + +``` +kubectl patch deploy neuvector-manager-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}' +kubectl patch deploy neuvector-scanner-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}' +kubectl patch cronjob neuvector-updater-pod -n cattle-neuvector-system --patch '{"spec":{"jobTemplate":{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}}}' +``` + +### 离线环境中的集群支持 + +- 所有 NeuVector 组件都可部署在离线环境中的集群上，无需任何额外配置。 + +### 支持限制 + +- 目前仅支持管理员和集群所有者。 + +- 不支持 Fleet 多集群部署。 + +- Windows 集群不支持 NeuVector。 + +### 其他限制 + +- 目前，如果 NeuVector partner Chart 已存在，则 NeuVector 功能 Chart 的安装会失败。要解决此问题，请卸载 NeuVector partner Chart 并重新安装 NeuVector 功能 Chart。 + +- Controller 未准备好时，有可能无法从 Rancher UI 访问 NeuVector UI。在此期间，Controller 将尝试重新启动，并且需要几分钟才能进入 active 状态。 + +- 安装 NeuVector Chart 时，不会针对不同的集群类型自动检测容器运行时。要解决此问题，你可以手动指定运行时。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/rancher-desktop.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/rancher-desktop.md new file mode 100644 index 00000000000..5e418069595 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/rancher-desktop.md @@ -0,0 +1,33 @@ +--- +title: 桌面上的 Kubernetes 与 Rancher Desktop +--- + + + + + +Rancher Desktop 将用于桌面开发和测试云原生应用程序的基本工具捆绑在一起。 + +如果你在本地计算机上运行适用于云环境的应用程序，则通常需要进行大量准备工作。你需要选择一个容器运行时,安装 Kubernetes 和流行的实用程序，并且可能还需要设置虚拟机。单独安装组件并使它们协同工作可能是一个耗时的过程。 + +为了降低复杂性，Rancher Desktop 为团队提供了以下主要功能： + +- 在 macOS、Linux 和 Windows 操作系统上进行简单易用的安装。 +- K3s，一个易于使用的轻量级 Kubernetes 发行版。 +- 能够在 Kubernetes 版本之间轻松切换。 +- 由 Rancher 支持的基于 GUI 的集群仪表板，用于浏览本地集群。 +- 自由选择容器引擎：dockerd（moby）或 containerd。 +- 用于配置应用程序以满足你的需求的首选项设置。 +- 容器、基于 Kubernetes 的开发和操作工作流程所需的捆绑工具。 +- 定期更新，使捆绑工具保持最新。 +- 与流行的工具/IDE 集成，包括 VS Code 和 Skaffold。 +- 镜像和注册表访问控制。 +- 支持 Docker 扩展。 + +访问 [Rancher Desktop](https://rancherdesktop.io) 网站并阅读[文档](https://docs.rancherdesktop.io/)以了解更多信息。 + +要在你的机器上安装 Rancher Desktop，请参阅[安装指南](https://docs.rancherdesktop.io/getting-started/installation)。 + +## 在 Rancher Desktop 上尝试 Rancher + +Rancher Desktop 提供了轻松试用容器化、基于 Helm 的应用程序所需的设置和工具。你可以按照此[操作指南](https://docs.rancherdesktop.io/how-to-guides/rancher-on-rancher-desktop)，使用 Rancher Desktop 开始使用 Rancher Kubernetes 管理平台。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8.json b/i18n/zh/docusaurus-plugin-content-docs/version-2.8.json index 2518b00a8d9..ebcffb97e2d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8.json +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8.json @@ -1,391 +1,402 @@ { - "version.label": { - "message": "v2.8", - "description": "The label for version 2.8" - }, - "sidebar.tutorialSidebar.category.Getting Started": { - "message": "开始使用", - "description": "The label for category Getting Started in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Quick Start Guides": { - "message": "快速入门指南", - "description": "The label for category Quick Start Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Deploy Rancher": { - "message": "部署 Rancher", - "description": "The label for category Deploy Rancher in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Deploy Workloads": { - "message": "部署工作负载", - "description": "The label for category Deploy Workloads in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Installation and Upgrade": { - "message": "安装和升级", - "description": "The label for category Installation and Upgrade in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Installation Requirements": { - "message": "安装要求", - "description": "The label for category Installation Requirements in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Installation References": { - "message": "安装参考", - "description": "The label for category Installation References in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Install/Upgrade on a Kubernetes Cluster": { - "message": "在 Kubernetes 集群上安装/升级", - "description": "The label for category Install/Upgrade on a Kubernetes Cluster in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Other Installation Methods": { - "message": "其他安装方式", - "description": "The label for category Other Installation Methods in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Air-Gapped Helm CLI Install": { - "message": "离线 Helm CLI 安装", - "description": "The label for category Air-Gapped Helm CLI Install in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Rancher on a Single Node with Docker": { - "message": "使用 Docker 将 Rancher 安装到单个节点中", - "description": "The label for category Rancher on a Single Node with Docker in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Rancher Behind an HTTP Proxy": { - "message": "HTTP 代理后的 Rancher", - "description": "The label for category Rancher Behind an HTTP Proxy in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Resources": { - "message": "资源", - "description": "The label for category Resources in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.How-to Guides": { - "message": "操作指南", - "description": "The label for category How-to Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.New User Guides": { - "message": "新用户指南", - "description": "The label for category New User Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Authentication, Permissions, and Global Configuration": { - "message": "认证、权限和全局配置", - "description": "The label for category Authentication, Permissions, and Global Configuration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Configuring Authentication": { - "message": "配置认证", - "description": "The label for category Authentication Config in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Configuring OpenLDAP": { - "message": "配置 OpenLDAP", - "description": "The label for category Configure OpenLDAP in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Configuring Microsoft AD Federation Service (SAML)": { - "message": "配置 Microsoft AD FS (SAML)", - "description": "The label for category Configure Microsoft AD Federation Service (SAML) in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Configuring Shibboleth (SAML)": { - "message": "配置 Shibboleth (SAML)", - "description": "The label for category Configure Shibboleth (SAML) in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Managing Role-Based Access Control (RBAC)": { - "message": "管理 RBAC", - "description": "The label for category Manage Role-Based Access Control (RBAC) in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.About Provisioning Drivers": { - "message": "关于 Provisioning 驱动", - "description": "The label for category About Provisioning Drivers in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.About RKE1 Templates": { - "message": "关于 RKE1 模板", - "description": "The label for category About RKE1 Templates in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Manage Clusters": { - "message": "管理集群", - "description": "The label for category Manage Clusters in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Access Clusters": { - "message": "访问集群", - "description": "The label for category Access Clusters in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Install Cluster Autoscaler": { - "message": "安装 Cluster Autoscaler", - "description": "The label for category Install Cluster Autoscaler in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Create Kubernetes Persistent Storage": { - "message": "创建 Kubernetes 持久存储", - "description": "The label for category Create Kubernetes Persistent Storage in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Provisioning Storage Examples": { - "message": "配置存储示例", - "description": "The label for category Provisioning Storage Examples in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Kubernetes Cluster Setup": { - "message": "Kubernetes 集群设置", - "description": "The label for category Kubernetes Cluster Setup in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Infrastructure Setup": { - "message": "基础设施设置", - "description": "The label for category Infrastructure Setup in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Kubernetes Clusters in Rancher Setup": { - "message": "Rancher 设置中的 Kubernetes 集群", - "description": "The label for category Kubernetes Clusters in Rancher Setup in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Checklist for Production-Ready Clusters": { - "message": "生产就绪集群检查清单", - "description": "The label for category Checklist for Production-Ready Clusters in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Set Up Clusters from Hosted Kubernetes Providers": { - "message": "通过托管 Kubernetes 提供商设置集群", - "description": "The label for category Set Up Clusters from Hosted Kubernetes Providers in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Use Windows Clusters": { - "message": "使用 Windows 集群", - "description": "The label for category Use Windows Clusters in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Set Up Cloud Providers": { - "message": "设置 Cloud Provider", - "description": "The label for category Set Up Cloud Providers in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Launch Kubernetes with Rancher": { - "message": "使用 Rancher 启动 Kubernetes", - "description": "The label for category Launch Kubernetes with Rancher in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Use New Nodes in an Infra Provider": { - "message": "在基础设施提供商中使用新节点", - "description": "The label for category Use New Nodes in an Infra Provider in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.vSphere": { - "message": "vSphere", - "description": "The label for category vSphere in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Nutanix": { - "message": "Nutanix", - "description": "The label for category Nutanix in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Kubernetes Resources Setup": { - "message": "Kubernetes 资源设置", - "description": "The label for category Kubernetes Resources Setup in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Workloads and Pods": { - "message": "工作负载和 Pod", - "description": "The label for category Workloads and Pods in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Horizontal Pod Autoscaler": { - "message": "Horizontal Pod Autoscaler", - "description": "The label for category Horizontal Pod Autoscaler in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Load Balancer and Ingress Controller": { - "message": "负载均衡和 Ingress Controller", - "description": "The label for category Load Balancer and Ingress Controller in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Helm Charts in Rancher": { - "message": "Rancher 中的 Helm Chart", - "description": "The label for category Helm Charts in Rancher in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Deploy Apps Across Clusters": { - "message": "跨集群部署应用", - "description": "The label for category Deploy Apps Across Clusters in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Backup, Restore, and Disaster Recovery": { - "message": "备份、还原和灾难恢复", - "description": "The label for category Backup, Restore, and Disaster Recovery in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Advanced User Guides": { - "message": "高级用户指南", - "description": "The label for category Advanced User Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Manage Projects": { - "message": "管理项目", - "description": "The label for category Manage Projects in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Manage Project Resource Quotas": { - "message": "管理项目资源配额", - "description": "The label for category Manage Project Resource Quotas in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Monitoring/Alerting Guides": { - "message": "Monitoring/Alerting 指南", - "description": "The label for category Monitoring/Alerting Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Prometheus Federator Guides": { - "message": "Prometheus Federator 指南", - "description": "The label for category Prometheus Federator Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Monitoring V2 Configuration Guides": { - "message": "Monitoring V2 配置指南", - "description": "The label for category Monitoring V2 Configuration Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Advanced Configuration": { - "message": "高级配置", - "description": "The label for category Advanced Configuration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Istio Setup Guide": { - "message": "Istio 设置指南", - "description": "The label for category Istio Setup Guide in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.CIS Scan Guides": { - "message": "CIS 扫描指南", - "description": "The label for category CIS Scan Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Enable Experimental Features": { - "message": "启用实验功能", - "description": "The label for category Enable Experimental Features in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Reference Guides": { - "message": "参考指南", - "description": "The label for category Reference Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Best Practices": { - "message": "最佳实践", - "description": "The label for category Best Practices in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Rancher Server": { - "message": "Rancher Server", - "description": "The label for category Rancher Server in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Rancher-Managed Clusters": { - "message": "Rancher 管理的集群", - "description": "The label for category Rancher-Managed Clusters in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Rancher Architecture": { - "message": "Rancher 架构", - "description": "The label for category Rancher Architecture in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Cluster Configuration": { - "message": "集群配置", - "description": "The label for category Cluster Configuration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Rancher Server Configuration": { - "message": "Rancher Server 配置", - "description": "The label for category Rancher Server Configuration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.GKE Cluster Configuration": { - "message": "GKE 集群配置", - "description": "The label for category GKE Cluster Configuration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Use Existing Nodes": { - "message": "使用现有节点", - "description": "The label for category Use Existing Nodes in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Downstream Cluster Configuration": { - "message": "下游集群配置", - "description": "The label for category Downstream Cluster Configuration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Node Template Configuration": { - "message": "节点模板配置", - "description": "The label for category Node Template Configuration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Machine Configuration": { - "message": "主机配置", - "description": "The label for category Machine Configuration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Single-Node Rancher in Docker": { - "message": "Docker 中的单节点 Rancher", - "description": "The label for category Single-Node Rancher in Docker in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Backup & Restore Configuration": { - "message": "备份和恢复配置", - "description": "The label for category Backup & Restore Configuration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Monitoring V2 Configuration": { - "message": "Monitoring V2 配置", - "description": "The label for category Monitoring V2 Configuration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Prometheus Federator": { - "message": "Prometheus Federator", - "description": "The label for category Prometheus Federator in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.User Settings": { - "message": "用户设置", - "description": "The label for category User Settings in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.CLI with Rancher": { - "message": "Rancher CLI", - "description": "The label for category CLI with Rancher in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.About the API": { - "message": "关于 API", - "description": "The label for category About the API in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Rancher Security": { - "message": "Rancher 安全", - "description": "The label for category Rancher Security in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.SELinux RPM": { - "message": "SELinux RPM", - "description": "The label for category SELinux RPM in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Integrations in Rancher": { - "message": "Rancher 中的集成", - "description": "The label for category Integrations in Rancher in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Cloud Marketplace Integration": { - "message": "云市场集成", - "description": "The label for category Cloud Marketplace Integration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.AWS Marketplace Integration": { - "message": "AWS Marketplace 集成", - "description": "The label for category AWS Marketplace Integration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.CIS Scans": { - "message": "CIS 扫描", - "description": "The label for category CIS Scans in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Istio": { - "message": "Istio", - "description": "The label for category Istio in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Configuration Options": { - "message": "配置选项", - "description": "The label for category Configuration Options in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Logging": { - "message": "Logging", - "description": "The label for category Logging in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Custom Resource Configuration": { - "message": "自定义资源配置", - "description": "The label for category Custom Resource Configuration in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Monitoring and Alerting": { - "message": "监控和告警", - "description": "The label for category Monitoring and Alerting in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.FAQ": { - "message": "常见问题", - "description": "The label for category FAQ in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Troubleshooting": { - "message": "故障排除", - "description": "The label for category Troubleshooting in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Kubernetes Components": { - "message": "Kubernetes 组件", - "description": "The label for category Kubernetes Components in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Other Troubleshooting Tips": { - "message": "其他故障排除提示", - "description": "The label for category Other Troubleshooting Tips in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Hardening Guides": { - "message": "加固指南", - "description": "The label for category Hardening Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.RKE Hardening Guides": { - "message": "RKE 加固指南", - "description": "The label for category RKE Hardening Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.RKE2 Hardening Guides": { - "message": "RKE2 加固指南", - "description": "The label for category RKE2 Hardening Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.K3s Hardening Guides": { - "message": "K3s 加固指南", - "description": "The label for category K3s Hardening Guides in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Continuous Delivery with Fleet": { - "message": "使用 Fleet 进行持续交付", - "description": "The label for category Continuous Delivery with Fleet in sidebar tutorialSidebar" - }, - "sidebar.tutorialSidebar.category.Example Workflows": { - "message": "工作流程示例", - "description": "The label for category Example Workflows in sidebar tutorialSidebar" - } + "version.label": { + "message": "v2.8", + "description": "The label for version 2.8" + }, + "sidebar.tutorialSidebar.category.Getting Started": { + "message": "开始使用", + "description": "The label for category Getting Started in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Quick Start Guides": { + "message": "快速入门指南", + "description": "The label for category Quick Start Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Deploy Rancher": { + "message": "部署 Rancher", + "description": "The label for category Deploy Rancher in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Deploy Workloads": { + "message": "部署工作负载", + "description": "The label for category Deploy Workloads in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Installation and Upgrade": { + "message": "安装和升级", + "description": "The label for category Installation and Upgrade in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Installation Requirements": { + "message": "安装要求", + "description": "The label for category Installation Requirements in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Installation References": { + "message": "安装参考", + "description": "The label for category Installation References in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Install/Upgrade on a Kubernetes Cluster": { + "message": "在 Kubernetes 集群上安装/升级", + "description": "The label for category Install/Upgrade on a Kubernetes Cluster in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Other Installation Methods": { + "message": "其他安装方式", + "description": "The label for category Other Installation Methods in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Air-Gapped Helm CLI Install": { + "message": "离线 Helm CLI 安装", + "description": "The label for category Air-Gapped Helm CLI Install in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Rancher on a Single Node with Docker": { + "message": "使用 Docker 将 Rancher 安装到单个节点中", + "description": "The label for category Rancher on a Single Node with Docker in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Rancher Behind an HTTP Proxy": { + "message": "HTTP 代理后的 Rancher", + "description": "The label for category Rancher Behind an HTTP Proxy in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Resources": { + "message": "资源", + "description": "The label for category Resources in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.How-to Guides": { + "message": "操作指南", + "description": "The label for category How-to Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.New User Guides": { + "message": "新用户指南", + "description": "The label for category New User Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Authentication, Permissions, and Global Configuration": { + "message": "认证、权限和全局配置", + "description": "The label for category Authentication, Permissions, and Global Configuration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Configuring Authentication": { + "message": "配置认证", + "description": "The label for category Authentication Config in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Configuring OpenLDAP": { + "message": "配置 OpenLDAP", + "description": "The label for category Configure OpenLDAP in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Configuring Microsoft AD Federation Service (SAML)": { + "message": "配置 Microsoft AD FS (SAML)", + "description": "The label for category Configure Microsoft AD Federation Service (SAML) in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Configuring Shibboleth (SAML)": { + "message": "配置 Shibboleth (SAML)", + "description": "The label for category Configure Shibboleth (SAML) in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Managing Role-Based Access Control (RBAC)": { + "message": "管理 RBAC", + "description": "The label for category Manage Role-Based Access Control (RBAC) in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.About Provisioning Drivers": { + "message": "关于 Provisioning 驱动", + "description": "The label for category About Provisioning Drivers in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.About RKE1 Templates": { + "message": "关于 RKE1 模板", + "description": "The label for category About RKE1 Templates in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Manage Clusters": { + "message": "管理集群", + "description": "The label for category Manage Clusters in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Access Clusters": { + "message": "访问集群", + "description": "The label for category Access Clusters in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Install Cluster Autoscaler": { + "message": "安装 Cluster Autoscaler", + "description": "The label for category Install Cluster Autoscaler in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Create Kubernetes Persistent Storage": { + "message": "创建 Kubernetes 持久存储", + "description": "The label for category Create Kubernetes Persistent Storage in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Provisioning Storage Examples": { + "message": "配置存储示例", + "description": "The label for category Provisioning Storage Examples in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Kubernetes Cluster Setup": { + "message": "Kubernetes 集群设置", + "description": "The label for category Kubernetes Cluster Setup in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Infrastructure Setup": { + "message": "基础设施设置", + "description": "The label for category Infrastructure Setup in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Kubernetes Clusters in Rancher Setup": { + "message": "Rancher 设置中的 Kubernetes 集群", + "description": "The label for category Kubernetes Clusters in Rancher Setup in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Checklist for Production-Ready Clusters": { + "message": "生产就绪集群检查清单", + "description": "The label for category Checklist for Production-Ready Clusters in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Set Up Clusters from Hosted Kubernetes Providers": { + "message": "通过托管 Kubernetes 提供商设置集群", + "description": "The label for category Set Up Clusters from Hosted Kubernetes Providers in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Use Windows Clusters": { + "message": "使用 Windows 集群", + "description": "The label for category Use Windows Clusters in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Set Up Cloud Providers": { + "message": "设置 Cloud Provider", + "description": "The label for category Set Up Cloud Providers in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Launch Kubernetes with Rancher": { + "message": "使用 Rancher 启动 Kubernetes", + "description": "The label for category Launch Kubernetes with Rancher in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Use New Nodes in an Infra Provider": { + "message": "在基础设施提供商中使用新节点", + "description": "The label for category Use New Nodes in an Infra Provider in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.vSphere": { + "message": "vSphere", + "description": "The label for category vSphere in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Nutanix": { + "message": "Nutanix", + "description": "The label for category Nutanix in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Kubernetes Resources Setup": { + "message": "Kubernetes 资源设置", + "description": "The label for category Kubernetes Resources Setup in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Workloads and Pods": { + "message": "工作负载和 Pod", + "description": "The label for category Workloads and Pods in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Horizontal Pod Autoscaler": { + "message": "Horizontal Pod Autoscaler", + "description": "The label for category Horizontal Pod Autoscaler in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Load Balancer and Ingress Controller": { + "message": "负载均衡和 Ingress Controller", + "description": "The label for category Load Balancer and Ingress Controller in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Helm Charts in Rancher": { + "message": "Rancher 中的 Helm Chart", + "description": "The label for category Helm Charts in Rancher in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Deploy Apps Across Clusters": { + "message": "跨集群部署应用", + "description": "The label for category Deploy Apps Across Clusters in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Backup, Restore, and Disaster Recovery": { + "message": "备份、还原和灾难恢复", + "description": "The label for category Backup, Restore, and Disaster Recovery in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Advanced User Guides": { + "message": "高级用户指南", + "description": "The label for category Advanced User Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Manage Projects": { + "message": "管理项目", + "description": "The label for category Manage Projects in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Manage Project Resource Quotas": { + "message": "管理项目资源配额", + "description": "The label for category Manage Project Resource Quotas in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Monitoring/Alerting Guides": { + "message": "Monitoring/Alerting 指南", + "description": "The label for category Monitoring/Alerting Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Prometheus Federator Guides": { + "message": "Prometheus Federator 指南", + "description": "The label for category Prometheus Federator Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Monitoring V2 Configuration Guides": { + "message": "Monitoring V2 配置指南", + "description": "The label for category Monitoring V2 Configuration Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Advanced Configuration": { + "message": "高级配置", + "description": "The label for category Advanced Configuration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Istio Setup Guide": { + "message": "Istio 设置指南", + "description": "The label for category Istio Setup Guide in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.CIS Scan Guides": { + "message": "CIS 扫描指南", + "description": "The label for category CIS Scan Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Enable Experimental Features": { + "message": "启用实验功能", + "description": "The label for category Enable Experimental Features in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Reference Guides": { + "message": "参考指南", + "description": "The label for category Reference Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Best Practices": { + "message": "最佳实践", + "description": "The label for category Best Practices in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Rancher Server": { + "message": "Rancher Server", + "description": "The label for category Rancher Server in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Rancher-Managed Clusters": { + "message": "Rancher 管理的集群", + "description": "The label for category Rancher-Managed Clusters in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Rancher Architecture": { + "message": "Rancher 架构", + "description": "The label for category Rancher Architecture in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Cluster Configuration": { + "message": "集群配置", + "description": "The label for category Cluster Configuration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Rancher Server Configuration": { + "message": "Rancher Server 配置", + "description": "The label for category Rancher Server Configuration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.GKE Cluster Configuration": { + "message": "GKE 集群配置", + "description": "The label for category GKE Cluster Configuration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Use Existing Nodes": { + "message": "使用现有节点", + "description": "The label for category Use Existing Nodes in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Downstream Cluster Configuration": { + "message": "下游集群配置", + "description": "The label for category Downstream Cluster Configuration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Node Template Configuration": { + "message": "节点模板配置", + "description": "The label for category Node Template Configuration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Machine Configuration": { + "message": "主机配置", + "description": "The label for category Machine Configuration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Single-Node Rancher in Docker": { + "message": "Docker 中的单节点 Rancher", + "description": "The label for category Single-Node Rancher in Docker in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Backup & Restore Configuration": { + "message": "备份和恢复配置", + "description": "The label for category Backup & Restore Configuration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Monitoring V2 Configuration": { + "message": "Monitoring V2 配置", + "description": "The label for category Monitoring V2 Configuration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Prometheus Federator": { + "message": "Prometheus Federator", + "description": "The label for category Prometheus Federator in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.User Settings": { + "message": "用户设置", + "description": "The label for category User Settings in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.CLI with Rancher": { + "message": "Rancher CLI", + "description": "The label for category CLI with Rancher in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.About the API": { + "message": "关于 API", + "description": "The label for category About the API in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Rancher Security": { + "message": "Rancher 安全", + "description": "The label for category Rancher Security in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.SELinux RPM": { + "message": "SELinux RPM", + "description": "The label for category SELinux RPM in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Integrations in Rancher": { + "message": "Rancher 中的集成", + "description": "The label for category Integrations in Rancher in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Cloud Marketplace Integration": { + "message": "云市场集成", + "description": "The label for category Cloud Marketplace Integration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.AWS Marketplace Integration": { + "message": "AWS Marketplace 集成", + "description": "The label for category AWS Marketplace Integration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.CIS Scans": { + "message": "CIS 扫描", + "description": "The label for category CIS Scans in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Istio": { + "message": "Istio", + "description": "The label for category Istio in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Configuration Options": { + "message": "配置选项", + "description": "The label for category Configuration Options in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Logging": { + "message": "Logging", + "description": "The label for category Logging in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Custom Resource Configuration": { + "message": "自定义资源配置", + "description": "The label for category Custom Resource Configuration in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Monitoring and Alerting": { + "message": "监控和告警", + "description": "The label for category Monitoring and Alerting in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.FAQ": { + "message": "常见问题", + "description": "The label for category FAQ in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Troubleshooting": { + "message": "故障排除", + "description": "The label for category Troubleshooting in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Kubernetes Components": { + "message": "Kubernetes 组件", + "description": "The label for category Kubernetes Components in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Other Troubleshooting Tips": { + "message": "其他故障排除提示", + "description": "The label for category Other Troubleshooting Tips in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Hardening Guides": { + "message": "加固指南", + "description": "The label for category Hardening Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.RKE Hardening Guides": { + "message": "RKE 加固指南", + "description": "The label for category RKE Hardening Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.RKE2 Hardening Guides": { + "message": "RKE2 加固指南", + "description": "The label for category RKE2 Hardening Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.K3s Hardening Guides": { + "message": "K3s 加固指南", + "description": "The label for category K3s Hardening Guides in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Continuous Delivery with Fleet": { + "message": "使用 Fleet 进行持续交付", + "description": "The label for category Continuous Delivery with Fleet in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Example Workflows": { + "message": "工作流程示例", + "description": "The label for category Example Workflows in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Virtualization on Kubernetes with Harvester": { + "message": "使用 Harvester 在 Kubernetes 上进行虚拟化", + "description": "The label for category Virtualization on Kubernetes with Harvester in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Cloud Native Storage with Longhorn": { + "message": "使用 Longhorn 进行云原生存储", + "description": "The label for category Cloud Native Storage with Longhorn in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Container Security with Neuvector": { + "message": "使用 NeuVector 实现容器安全", + "description": "The label for category Container Security with Neuvector in sidebar tutorialSidebar" } - \ No newline at end of file +} diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/aws-cloud-marketplace.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/aws-cloud-marketplace.md new file mode 100644 index 00000000000..c7dd9264ee4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/aws-cloud-marketplace.md @@ -0,0 +1,36 @@ +--- +title: AWS Marketplace 集成 +--- + + + + + +## 概述 + +Rancher 提供与 AWS Marketplace 的集成，允许用户购买 SUSE 的支持合同。当你开始支持更多集群时，这种集成使你可以轻松调整支持需求。 + +## 限制 + +- 你必须运行 Rancher v2.6.7 或更高版本。 +- Rancher 的部署必须启用额外的指标。 +- Rancher 必须安装在 EKS 集群上。 +- 你必须通过 AWS Marketplace 购买至少一项 Rancher 支持的权限。 +- 你可能需要额外的设置来支持代理/离线用例。有关更多信息，请参阅[先决条件](adapter-requirements.md)。 + +## 如何使用 + +1. 完成[先决条件](adapter-requirements.md)。 +2. [安装 CSP Adapter](install-adapter.md)。 + +## 常见问题 + +**我以后可以购买对更多节点的支持吗？** + +可以。只需转到你最初用于购买支持的 AWS Marketplace 条目并增加权限数量即可。 + +**我可以在同一个 AWS 账户中使用多个 Rancher 实例吗？** + +可以。但是，Rancher 安装的每个集群都需要遵守先决条件。 + +此外，一个给定的权限一次只能由一个 Rancher 管理服务器使用。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cloud-marketplace/cloud-marketplace.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cloud-marketplace/cloud-marketplace.md new file mode 100644 index 00000000000..44c95be8fe9 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cloud-marketplace/cloud-marketplace.md @@ -0,0 +1,11 @@ +--- +title: 云市场集成 +--- + + + + + +Rancher 提供与云市场的集成，以便轻松购买对托管在某些云提供商上的安装支持。此外，此集成还提供了生成 Supportconfig Bundle 的功能，该 Bundle 可以给 rancher 提供支持。 + +此集成仅支持 AWS。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/elemental/elemental.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/elemental/elemental.md new file mode 100644 index 00000000000..6352de02ba1 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/elemental/elemental.md @@ -0,0 +1,30 @@ +--- +title: 使用 Elemental 进行操作系统管理 +--- + + + + + +Elemental 支持云原生主机管理。Elemental 允许你在任何位置装载任何机器，无论是在数据中心还是在边缘，并将它们无缝集成到 Kubernetes 中，同时管理你的工作流程（例如操作系统更新）。 + +## Elemental 和 Rancher + +Rancher 中的 Elemental： + +- 是 Kubernetes 原生的，它允许你通过 Kubernetes 集群中的 Elemental 管理操作系统。 +- 从 Kubernetes 的操作角度来看，不会造成干扰。 +- 是声明性的，并且 GitOps 友好。 +- 允许可信的、确定性的和可预测的 OCI Image-based flows。 +- 大规模运作。它支持 Fleet 规模的操作系统管理。 + +### 我什么时候应该使用 Elemental？ + +- Elemental 通过 Rancher manager 实现云原生操作系统管理。它适用于任何操作系统（例如，SLE Micro vanilla）。 +- Elemental 允许对数据中心和边缘的机器进行云原生管理。 +- Elemental 非常灵活，允许平台团队在其机器群中执行各种工作流。 + +## Elemental 和 Rancher Prime + +- 已作为 GUI 扩展深度集成到 Rancher 中。 +- 将 Rancher 的用例扩展到操作系统，如今 SLE Micro 与 Rancher 可以完美配合。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/architecture.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/architecture.md new file mode 100644 index 00000000000..ffdbff3edc2 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/architecture.md @@ -0,0 +1,11 @@ +--- +title: 架构 +--- + + + + + +Fleet 可以管理来自 Git 的原始 Kubernetes YAML、Helm Chart、Kustomize 或三者的任何组合的部署。无论来源如何，所有资源都会动态转化为 Helm Chart，并使用 Helm 作为引擎来部署到集群中的所有内容。这为你提供了高度的控制力、一致性和可审计性。Fleet 不仅关注扩展能力，而且还提供高度的控制和可见性，从而让用户准确了解集群上安装的内容。 + +![架构](/img/fleet-architecture.svg) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/fleet.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/fleet.md new file mode 100644 index 00000000000..8b1249e41a0 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/fleet.md @@ -0,0 +1,27 @@ +--- +title: 使用 Fleet 进行持续交付 +--- + + + + + +Fleet 通过集群 Fleet 的供应链协调和管理应用程序的持续交付。Fleet 使用 GitOps 作为一种安全的运营模式，组织供应链，帮助团队及时自信地交付。 + +## Fleet 和 Rancher + +许多用户经常同时管理 10 个以上的集群。鉴于集群的激增，持续交付是 Rancher 的重要组成部分。Fleet 使用 GitOps 确保可靠的持续交付体验，这是一种安全且越来越常见的运营模式。 + +### 我什么时候应该使用 Fleet? + +- 我需要跨地理区域部署我的监控堆栈（例如 Grafana、Prometheus），每个区域都有不同的保留策略。 +- 我是一名平台运营商，希望使用可扩展且安全的操作模型（GitOps）为集群配置所有组件。 +- 我是一名应用程序开发人员，希望我的最新更改自动进入我的开发环境。 + +## Fleet 和 Rancher Prime + +Fleet 已经作为持续交付工具和 GitOps 引擎深度集成到 Rancher 中。 + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/overview.md new file mode 100644 index 00000000000..55b89b8ee08 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/overview.md @@ -0,0 +1,66 @@ +--- +title: 概述 +--- + + + + + +使用 Fleet 进行持续交付是大规模的 GitOps。Fleet 旨在管理多达一百万个集群。它也足够轻量级，对于[单个集群](https://fleet.rancher.io/installation#default-install)也很有效，但当你达到[大规模](https://fleet.rancher.io/installation#configuration-for-multi-cluster)时，它真的会大放异彩。大规模是指单个组织中的大量集群、大量部署或大量团队。 + +Fleet 是 Rancher 的一个独立项目，可以通过 Helm 安装在任何 Kubernetes 集群上。 + +## 架构 + +有关 Fleet 如何运作的信息，请参阅[架构](./architecture)页面。 + +## 在 Rancher UI 中访问 Fleet + +Fleet 预安装在 Rancher 中，并由 Rancher UI 中的**持续交付**选项进行管理。有关持续交付和其他 Fleet 故障排除技巧的更多信息，请参阅[此处](https://fleet.rancher.io/troubleshooting)。 + +用户可以按照 **gitops** 实践，利用持续交付将应用程序部署到 git 仓库中的 Kubernetes 集群，而无需任何手动操作。 + +按照以下步骤在 Rancher UI 中访问持续交付： + +1. 单击 **☰ > 持续交付**。 + +1. 在菜单顶部选择命名空间 ，注意以下事项： + + - 默认情况下，选择 **fleet-default**，包括通过 Rancher 注册的所有下游集群。 + + - 你可以切换到 **fleet-local**，它仅包含 **local** 集群，或者你可以创建自己的工作空间，将集群分配和移动到其中。 + - 然后，你可以通过单击左侧导航栏上的**集群**来管理集群。 + +1. 单击左侧导航栏上的 **Gitrepos**，将 gitrepo 部署到当前工作空间的集群中。 + +1. 选择 [git 仓库](https://fleet.rancher.io/gitrepo-add)和[目标集群/集群组](https://fleet.rancher.io/gitrepo-targets)。你也可以通过单击左侧导航栏中的**集群组**在 UI 中创建集群组。 + +1. 部署 gitrepo 后，你可以通过 Rancher UI 监控应用程序。 + +## Windows 支持 + +有关对具有 Windows 节点的集群的支持的详细信息，请参阅 [Windows 支持](./windows-support)页面。 + +## GitHub 仓库 + +Fleet Helm charts 可在[此处](https://github.com/rancher/fleet/releases)获取。 + +## 在代理后使用 Fleet + +有关在代理后面使用 Fleet 的详细信息，请参阅[在代理后使用 Fleet](./use-fleet-behind-a-proxy)页面。 + +## Helm Chart 依赖 + +为了成功部署具有依赖项的 Helm Chart，你必须运行一个手动命令（如下所示），因为用户需要满足依赖列表。如果不执行此操作，继续克隆你的仓库并运行 `helm install`，则会因依赖项丢失而导致安装失败。 + +git 仓库中的 Helm Chart 必须在 Chart 子目录中包含其依赖。 你必须手动运行 `helm dependencies update $chart` 或在本地运行 `helm dependencies build $chart`，然后将完整的 Chart 目录提交到 git 仓库。请注意，你需要使用适当的参数更新你的命令。 + +## 故障排除 + +- **已知问题**：Fleet gitrepos 的 clientSecretName 和 helmSecretName 密文不包含在 [backup-restore-operator](../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md#1-install-the-rancher-backup-operator) 创建的备份或恢复中。一旦有永久的解决方案，我们将更新社区内容。 + +- **临时解决方法**：默认情况下，用户定义的密文不会在 Fleet 中备份。如果执行灾难恢复或将 Rancher 迁移到新集群，则有必要重新创建密文。要修改 ResourceSet 以包含要备份的额外资源，请参阅文档[此处](https://github.com/rancher/backup-restore-operator#user-flow)。 + +## 文档 + +Fleet 文档位于 https://fleet.rancher.io/ 。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md new file mode 100644 index 00000000000..3806cbb2248 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md @@ -0,0 +1,58 @@ +--- +title: 在代理后使用 Fleet +--- + + + + + +在本节中，你将学习如何在具有公共 IP 的 Rancher 服务器，以及一个没有公共 IP 但配置为使用代理的 Kubernetes 集群的设置中启用 Fleet。 + +Rancher 不会与已注册的下游集群建立连接。部署在下游集群上的 Rancher agent 必须能够与 Rancher 建立连接。 + +要让 Fleet 在代理后工作，你需要为下游集群设置 **Agent 环境变量**。以下是集群级别的配置选项。 + +你可以通过 Rancher UI 为任何集群类型（包括注册集群和自定义集群）配置这些环境变量。可以在编辑现有集群或配置新集群时添加变量。 + +对于公共下游集群，[在 Rancher UI 中设置必要的环境变量](#在-rancher-ui-中设置环境变量)就足够了。 + +对于私有节点或私有集群，则需要在节点上设置环境变量。然后，在配置自定义集群或注册私有集群时，在 Rancher UI 中配置环境变量。有关如何在 K3s 集群中的 Ubuntu 节点上设置环境变量的示例，请参阅[本节](#在私有节点上设置环境变量)。 + +## 必要的环境变量 + +为代理添加 Fleet agent 环境变量时，将 替换为你的私有代理 IP。 + +| 变量名称 | 值 | +| ------------- | ----------------------------------------------------------------------- | +| `HTTP_PROXY` | http://:8888 | +| `HTTPS_PROXY` | http://:8888 | +| `NO_PROXY` | 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local | + +## 在 Rancher UI 中设置环境变量 + +要将环境变量添加到现有集群： + +1. 单击 **☰ > 集群管理**。 +1. 转到要添加环境变量的集群，然后单击 **⋮ > 编辑配置**。 +1. 单击**高级选项**。 +1. 单击**添加环境变量**。 +1. 输入[必要的环境变量](#必要的环境变量) +1. 单击**保存**。 + +**结果：** Fleet agent 会在代理后工作。 + +## 在私有节点上设置环境变量 + +对于私有节点和私有集群，需要在节点本身上设置代理环境变量，并在 Rancher UI 中进行配置。 + +此示例显示了如何在 K3s 集群中的 Ubuntu 节点上设置环境变量： + +``` +ssh -o ForwardAgent=yes ubuntu@ +ssh +export proxy_private_ip= +export HTTP_PROXY=http://${proxy_private_ip}:8888 +export HTTPS_PROXY=http://${proxy_private_ip}:8888 +export NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/windows-support.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/windows-support.md new file mode 100644 index 00000000000..7102f0fa01d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/fleet/windows-support.md @@ -0,0 +1,25 @@ +--- +title: Windows 支持 +--- + + + + + +在 Rancher v2.5.6 之前，`agent` 在具有 Windows 节点的下游集群上没有原生的 Windows 清单。这将导致集群的 `agent` pod 运行失败。 + +如果你从旧版本的 Rancher 升级到 v2.5.6+，你可以在 _下游集群_ 中部署具有以下工作流的可运行的 `agent`： + +1. 封锁所有 Windows 节点。 +1. 对 `agent` 工作负载应用以下容忍度。 +1. 取消所有 Windows 节点的封锁。 +1. 删除所有 `agent` pod。使用新的容忍度来创建新 pod。 +1. 一旦 `agent` pod 运行，并且 Fleet 的自动更新已启用，它们会更新到兼容 Windows 的 `agent` 版本。 + +```yaml +tolerations: + - effect: NoSchedule + key: cattle.io/os + operator: Equal + value: linux +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/harvester/harvester.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/harvester/harvester.md new file mode 100644 index 00000000000..8f4b2312b56 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/harvester/harvester.md @@ -0,0 +1,15 @@ +--- +title: 使用 Harvester 在 Kubernetes 上进行虚拟化 +--- + + + + + +## Harvester + +Harvester 是 Rancher v2.6.1 新增的功能，是基于 Kubernetes 构建的开源超融合基础架构（HCI）软件。Harvester 安装在裸金属服务器上，提供集成的虚拟化和分布式存储功能。虽然 Harvester 使用 Kubernetes 运行，但它不需要用户了解 Kubernetes 概念，这使得它更加用户友好。 + +## Harvester 与 Rancher + +凭借 Rancher Prime 和 Harvester，IT 运维人员现在可以访问一个企业级的、易于使用的基础设施平台，该平台可以协同管理他们的虚拟机和 Kubernetes 集群。有关产品支持的更多信息，请参阅[支持矩阵](https://www.suse.com/suse-harvester/support-matrix/all-supported-versions/harvester-v1-2-0/)。通过 Rancher 虚拟化管理功能，用户可以导入和管理多个 Harvester 集群。利用 Rancher 的认证功能和 RBAC 控制来支持多租户。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/harvester/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/harvester/overview.md new file mode 100644 index 00000000000..ac720bec060 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/harvester/overview.md @@ -0,0 +1,42 @@ +--- +title: 概述 +--- + + + + + +[Harvester](https://docs.harvesterhci.io/) 是 Rancher v2.6.1 新增的功能，是基于 Kubernetes 构建的开源超融合基础架构（HCI）软件。Harvester 安装在裸金属服务器上，提供集成的虚拟化和分布式存储功能。虽然 Harvester 使用 Kubernetes 运行，但它不需要用户了解 Kubernetes 概念，这使得它更加用户友好。 + +### 功能开关 + +Harvester 功能开关用于管理对 Rancher 中虚拟化管理（VM）页面的访问，用户可以直接导航到 Harvester 集群并访问 Harvester UI。Harvester 的功能开关默认启用。如需了解 Rancher 中功能开关的更多详细信息，请单击[此处](../../how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 + +要导航到 Harvester 集群，请单击 **☰ > 虚拟化管理**。在 Harvester 集群页面中，单击集群以转到该 Harvester 集群的视图。 + +- 如果启用了 Harvester 功能开关，Harvester 集群将会从列出 Kubernetes 集群的任何页面或应用（例如 Fleet 和多集群应用）中过滤掉。 + +- 如果禁用了 Harvester 功能开关，并且导入了 Harvester 集群，Harvester 集群将显示在集群管理页面的 Rancher 集群列表中。仅当功能开关为关闭时，Harvester 集群才会显示在集群列表中。 + +- 集成 Harvester 后，你可以将 Harvester 集群导入 Rancher，对应的集群类型是 `Harvester`。 + +- 用户只能在虚拟化管理页面上导入 Harvester 集群。不支持在集群管理页面上导入集群，并且会出现警告，建议你返回虚拟化管理页面执行此操作。 + +### Harvester 主机驱动 + +[Harvester 主机驱动](https://docs.harvesterhci.io/v1.1/rancher/node/node-driver/)通常可用于 Rancher 中的 RKE 和 RKE2 选项。无论 Harvester 功能开关是否启用，主机驱动都是可用的。请注意，主机驱动默认处于关闭状态。用户只能通过集群管理页面在 Harvester 上创建 RKE 或 RKE2 集群。 + +Harvester 允许通过 Harvester UI 上传和显示 `.ISO` 镜像，但 Rancher UI 是不支持的。这是因为 `.ISO` 镜像通常需要额外的设置，这会干扰干净的部署（即无需用户干预），并且它们通常不用于云环境。 + +如需了解 Rancher 中主机驱动的更多详细信息，请单击[此处](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/about-provisioning-drivers#主机驱动)。 + +### 端口要求 + +Harvester 集群的端口要求可以在[此处](https://docs.harvesterhci.io/v1.1/install/requirements#networking)找到。 + +此外，其他网络注意事项如下： + +- 请务必为 VM VLAN 网络启用物理交换机的 VLAN 中继端口。 +- 按照[此处](https://docs.harvesterhci.io/v1.1/networking/index)的网络设置指南进行操作。 + +对于其他集群（例如 K3s 和 RKE1）的其他端口要求，请参阅[这些文档](https://docs.harvesterhci.io/v1.1/install/requirements/#guest-clusters)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/integrations-in-rancher.mdx b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/integrations-in-rancher.mdx new file mode 100644 index 00000000000..c0a824cc9da --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/integrations-in-rancher.mdx @@ -0,0 +1,51 @@ +--- +title: Rancher 中的集成 +--- + + + + + +import { Card, CardSection } from "@site/src/components/CardComponents"; +import { RocketRegular } from "@fluentui/react-icons"; + +Prime 是 Rancher 生态系统的企业级产品，具有更高的安全性、更长的生命周期和对 Prime 专有文档的访问权限。Rancher Prime 安装资产托管在受信任的 SUSE 注册表上，由 Rancher 拥有和管理。受信任的 Prime 注册表仅包括经过社区测试的稳定版本。 + +Prime 还提供生产支持选项，以及根据你的商业需求定制的订阅附加组件。 + +要了解更多信息并开始使用 Rancher Prime，请访问[本页](https://www.rancher.com/quick-start)。 + +}> + + + + + + + + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/kubernetes-distributions/kubernetes-distributions.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/kubernetes-distributions/kubernetes-distributions.md new file mode 100644 index 00000000000..b45cce6b4d7 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/kubernetes-distributions/kubernetes-distributions.md @@ -0,0 +1,34 @@ +--- +title: Kubernetes 发行版 +--- + + + + + +## K3s + +K3s 是一款轻量级、完全兼容的 Kubernetes 发行版，专为一系列用例设计，包括边缘计算、物联网、CI/CD、开发和将 Kubernetes 嵌入到应用程序中。它将系统打包为单个二进制文件，使用 sqlite3 作为默认存储，并提供用户友好的启动器，从而简化了 Kubernetes 的管理。K3s 包括 Local Storage 和负载均衡、Helm Chart 控制器和 Traefik CNI 等基本功能。它最大限度地减少了外部依赖，并提供了精简的 Kubernetes 体验。K3s 于 2020 年 6 月作为沙箱项目捐赠给 CNCF。 + +### K3s 与 Rancher + +- Rancher 允许在一系列平台上轻松配置 K3s，包括 Amazon EC2、DigitalOcean、Azure、vSphere 或现有服务器。 +- Kubernetes 集群的标准 Rancher 管理，包括所有概述[集群管理功能](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup#cluster-management-capabilities-by-cluster-type)。 + +## RKE2 + +RKE2 是 Rancher 开发的一款兼容 Kubernetes 的发行版。它是专门为美国联邦政府部门的安全性和合规性而设计的。 + +RKE2 的主要特性包括: + +1. **安全性和合规性重点**：RKE2 非常重视安全性和合规性，在“默认安全”框架下运行，适用于政府服务以及金融和医疗保健等高度监管的行业。 +1. **CIS Kubernetes Benchmark 一致性**：RKE2 经过预配置，符合 CIS Kubernetes Hardening Benchmark（目前支持 v1.23 和 v1.7），并且只需最少的手动干预。 +1. **FIPS 140-2 合规性**：RKE2 符合 FIPS 140-2 标准，使用经过 FIPS 验证的加密模块作为其组件。 +1. **嵌入式 ETCD**：RKE2 默认使用嵌入式 ETCD 作为其数据存储。这使其与标准的 Kubernetes 实践更加紧密地结合在一起，从而可以更好地与其他 Kubernetes 工具集成并降低错误配置的风险。 +1. **与上游 Kubernetes 保持一致**：RKE2 旨在与上游 Kubernetes 保持密切一致，降低在使用偏离标准 Kubernetes 实践的发行版时可能出现的不一致风险。 +1. **多重 CNI 支持**：RKE2 支持多个容器网络接口（CNI）插件，包括 Cilium、Calico 和 Multus。这对于具有各种生产设施的电信分发中心和工厂等用例至关重要。 + +## RKE2 与 Rancher + +- Rancher 允许在一系列平台上轻松配置 RKE2，包括 Amazon EC2、DigitalOcean、Azure、vSphere 或现有服务器。 +- Kubernetes 集群的标准 Rancher 管理，包括所有概述[集群管理功能](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup#cluster-management-capabilities-by-cluster-type)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/kubewarden/kubewarden.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/kubewarden/kubewarden.md new file mode 100644 index 00000000000..76f8c16d82a --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/kubewarden/kubewarden.md @@ -0,0 +1,31 @@ +--- +title: 使用 Kubewarden 进行高级策略管理 +--- + + + + + +Kubewarden 是一个策略引擎，用于保护和帮助管理集群资源。它允许通过策略验证和更改资源请求，包括上下文感知策略和验证镜像签名。它可以在监视或强制模式下运行策略，并提供集群状态的概述。 + +Kubewarden 旨在通过启用和简化“策略即代码”来成为通用策略引擎。Kubewarden 策略被编译到 WebAssembly 中： 它们体积小（400KB~2MB），沙箱式，安全且便携。它旨在通过迎合组织中的每个人来实现通用性： + +- 策略用户：使用 Kubernetes 自定义资源管理和声明策略，重新使用 Rego（OPA 和 Gatekeeper）中编写的现有策略。在 CI/CD 中测试群集外部的策略。 +- 策略开发人员：使用你喜欢的 Wasm 编译语言（Rego、Go、Rust、C#、Swift、Typescript 等）编写策略。重新使用你已经熟悉的工具、库和工作流的生态系统。 +- 策略分发器：策略是 OCI 工件，通过 OCI 仓库为它们提供服务，并在你的基础设施中使用行业标准，如 Software-Bill-Of-Materials 和工件签名。 +- 集群操作员：Kubewarden 是模块化的（OCI 注册表、PolicyServers、Audit Scanner、Controller）。配置你的部署以满足你的需求，隔离不同的租户。使用审核扫描程序和策略报告在集群中获取过去、当前和可能的违规行为的概述。 +- Kubewarden Integrator：将其用作编写新的 Kubewarden 模块和自定义策略的平台。 + +## Kubewarden 和 Rancher + +Kubewarden 的上游 Helm Chart 完全集成为 Rancher 应用程序，为安装选项提供了 UI。这些 Chart 还附带了尊重 Rancher 堆栈的默认设置（例如：不监管 Rancher 系统命名空间）以及默认的 PolicyServer 和策略。用户可以访问所有 Kubewarden 功能，并可以通过与 Kubernetes API 交互（例如：使用 kubectl）手动部署 PolicyServer 和策略。 + +Kubewarden 提供了对已删除的 Kubernetes Pod 安全策略的完全替换。 Kubewarden 还通过增强其安全功能，与最新版本的 Kubernetes 引入的新 Pod Security Admission 功能进行了集成。 + +## Kubewarden 和 Rancher Prime + +Kubewarden 的 Rancher UI 扩展将其集成到 Rancher UI 中。UI 扩展可自动安装和配置 Kubewarden 堆栈，并配置对 SUSE 维护的策略的访问。UI 扩展提供了对现成策略的策划目录的访问。使用 UI 扩展，人们可以浏览、安装和配置这些策略。 + +UI 扩展提供了 Kubewarden 堆栈组件及其行为的概述。这包括对 Kubewarden 指标和跟踪事件的访问。操作员可以了解策略对集群的影响并排查问题。 + +此外，UI 扩展还提供了 Policy Reporter UI，它可以直观地概述 Kubernetes 集群的合规性状态。通过此 UI，操作员可以快速识别所有不合规的 Kubernetes 资源，了解违规原因并采取相应行动。所有这一切都得益于 Rancher Prime 的支持。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/longhorn/longhorn.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/longhorn/longhorn.md new file mode 100644 index 00000000000..c3727f30ff8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/longhorn/longhorn.md @@ -0,0 +1,15 @@ +--- +title: 使用 Longhorn 进行云原生存储 +--- + + + + + +## Longhorn + +Longhorn 是官方的[云原生计算基金会（CNCF）](https://cncf.io/)项目，它为 Kubernetes 提供了一个可以在任何地方运行的强大的云原生分布式存储平台。当与 Rancher 相结合使用时，Longhorn 使你可以在 Kubernetes 环境中轻松、快速且可靠地部署高可用的持久块存储。 + +## Longhorn 与 Rancher + +凭借 Rancher Prime 和 Longhorn，用户可以通过 Rancher 应用商店一键轻松部署，并对托管集群进行生命周期管理；允许用户能够安装和升级，同时进行清空操作以实现优雅的操作。Longhorn 和 Rancher Prime 还提供了 Windows 的混合集群支持，Rancher 托管的镜像，通过 Rancher 的 UI 代理访问，以及使用 Longhorn 指标进行 Rancher 监控。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/longhorn/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/longhorn/overview.md new file mode 100644 index 00000000000..b5298c3c21d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/longhorn/overview.md @@ -0,0 +1,74 @@ +--- +title: 概述 +--- + + + + + +[Longhorn](https://longhorn.io/) 是一个轻量级、可靠且易于使用的 Kubernetes 分布式块存储系统。 + +Longhorn 是免费的开源软件。它最初由 Rancher Labs 开发，现在被开发为云原生计算基金会的沙箱项目。它可以通过 Helm、kubectl 或 Rancher UI 安装在任何 Kubernetes 集群上。有关其架构的更多信息，请参阅[此处](https://longhorn.io/docs/latest/concepts/)。 + +使用 Longhorn，你可以： + +- 使用 Longhorn 卷作为 Kubernetes 集群中分布式有状态应用程序的持久存储 +- 将块存储分区为 Longhorn 卷，以便你可以在有或没有云提供商的情况下使用 Kubernetes 卷 +- 跨多个节点和数据中心复制块存储以提高可用性 +- 将备份数据存储在 NFS 或 AWS S3 等外部存储中 +- 创建跨集群灾难恢复卷，以便使用另一个 Kubernetes 集群中的备份快速恢复主 Kubernetes 集群中的数据 +- 计划卷的定期快照，并将定期备份调度到 NFS 或兼容 S3 的辅助存储 +- 使用备份来恢复卷 +- 在不中断持久卷的情况下升级 Longhorn + +
Longhorn 仪表板
+ +![Longhorn 仪表板](/img/longhorn-screenshot.png) + +### 使用 Rancher 安装 Longhorn + +1. 满足所有[安装要求](https://longhorn.io/docs/latest/deploy/install/#installation-requirements)。 +1. 转到要安装 Longhorn 的集群。 +1. 单击 **Apps**。 +1. 单击 **Charts**。 +1. 单击 **Longhorn**。 +1. 可选：要自定义初始设置，请单击 **Longhorn 默认设置**并编辑配置。如需自定义设置的帮助，请参阅 [Longhorn 文档](https://longhorn.io/docs/latest/references/settings/)。 +1. 单击**安装**。 + +**结果**：Longhorn 已部署到 Kubernetes 集群中。 + +### 从 Rancher UI 访问 Longhorn + +1. 转到安装了 Longhorn 的集群。在左侧导航菜单中，单击 **Longhorn**。 +1. 在此页面上，你可以编辑 Longhorn 管理的 Kubernetes 资源。要查看 Longhorn UI，请单击**概述**中的 **Longhorn** 按钮。 + +**结果**：你将转到 Longhorn UI，在这里你可以管理 Kubernetes 集群中的 Longhorn 卷及其副本，以及可能存在于另一个 Kubernetes 集群或 S3 中的 Longhorn 存储辅助备份。 + +### 从 Rancher UI 卸载 Longhorn + +1. 转到安装了 Longhorn 的集群，然后单击 **Apps**。 +1. 点击**已安装的应用**。 +1. 转到 `longhorn-system` 命名空间并选中 `longhorn` 和 `longhorn-crd` 应用程序旁边的框。 +1. 单击**删除**并确认**删除**。 + +**结果**：Longhorn 已被卸载。 + +### GitHub 仓库 + +Longhorn 项目可在[此处](https://github.com/longhorn/longhorn)获取。 + +### 文档 + +Longhorn 文档在[此处](https://longhorn.io/docs/)。 + +### 架构 + +Longhorn 为每个卷创建专用的存储控制器，并在多个节点上存储的多个副本之间同步复制该卷。 + +存储控制器和副本本身是使用 Kubernetes 编排的。 + +有关其架构的更多信息，请参阅[此处](https://longhorn.io/docs/latest/concepts/)。 + +
Longhorn 架构
+ +![Longhorn 架构](/img/longhorn-architecture.svg) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/neuvector/neuvector.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/neuvector/neuvector.md new file mode 100644 index 00000000000..d2f1257309f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/neuvector/neuvector.md @@ -0,0 +1,27 @@ +--- +title: 使用 NeuVector 实现容器安全 +--- + + + + + +NeuVector 是唯一一个100%开源、零信任的容器安全平台。在整个容器生命周期中持续扫描。清除安全路障。从一开始就制定安全策略，以最大限度地提高开发人员的灵活性。NeuVector 提供从构建到生产的漏洞和合规性扫描与管理。独特的 NeuVector 运行时保护通过七层容器防火墙保护集群内的网络连接以及集群的入口/出口。此外，NeuVector 还监视容器和主机上的进程和文件活动，以阻止未经授权的活动。 + +## NeuVector 和 Rancher + +所有 NeuVector 功能均可通过 Rancher 进行集成部署并单点登录 NeuVector 控制台。Rancher 集群管理员能够在其集群上部署和管理 NeuVector 部署，Helm 值、configMaps、自定义资源定义（CRD）和 NeuVector 控制台轻松配置 NeuVector。 + +使用 NeuVector 和 Rancher： + +- 部署、管理和保护多个集群。 +- 管理和报告 Rancher 工作负载和节点的漏洞和合规性结果。 + +## NeuVector Prime 和 Rancher Prime + +Rancher Manager 的 NeuVector UI 扩展可用于并支持 Rancher Prime 和 NeuVector Prime 客户。此扩展提供： + +- NeuVector 自动化部署，包括 Rancher Prime NeuVector 扩展仪表板。 +- 访问每个集群的重要安全信息，如关键安全事件、漏洞扫描结果和入口/出口暴露。 +- 直接对 Rancher 资源（例如节点和容器/ Pod）进行集成漏洞 (CVE) 和合规扫描。 +- 集成操作，如手动触发 Rancher 资源的扫描。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/neuvector/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/neuvector/overview.md new file mode 100644 index 00000000000..2a23c695335 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/neuvector/overview.md @@ -0,0 +1,196 @@ +--- +title: 概述 +--- + + + + + +### Rancher 中的 NeuVector 集成 + +[NeuVector 5.x](https://open-docs.neuvector.com/) 是一个开源的，以容器为中心的安全应用程序，Rancher 已集成 NeuVector。NeuVector 在运行时为关键应用程序和数据提供实时的合规、可见和保护功能。NeuVector 提供具有 CIS Benchmark 和漏洞扫描的防火墙、容器进程/文件系统监控和安全审计。有关 Rancher 安全性的更多信息，请参阅[安全文档](../../reference-guides/rancher-security)。 + +NeuVector 可以通过 Helm Chart 启用。你可以在 **Apps** 或 Rancher UI 中的 **Cluster Tools** 中安装该 Chart。安装 Helm Chart 后，用户可以轻松地[在 Rancher 中部署和管理 NeuVector 集群](https://open-docs.neuvector.com/deploying/rancher#deploy-and-manage-neuvector-through-rancher-apps-marketplace)。 + +### 使用 Rancher 安装 NeuVector + +Harvester Helm Chart 用于管理 Rancher 中 NeuVector UI 的访问，用户可以在 Rancher 中直接跳转，然后部署和管理 NeuVector 集群。 + +**通过 Apps 导航并安装 NeuVector Chart：** + +1. 单击 **☰ > 集群管理**。 +1. 在 Clusters 页面上，转到要部署 NeuVector 的集群，然后单击 **Explore**。 +1. 转到 **Apps > Charts**，然后从 Chart 仓库中安装 **NeuVector**。 +1. 不同的集群类型需要不同的容器运行时。配置 Helm Chart 值时，转到**容器运行时**，然后根据集群类型选择运行时。最后，再次单击**安装**。 + +以下是一些例子： + +- RKE1：`docker` +- K3s 和 RKE2：`k3scontainerd` +- AKS：`containerd` 适用于 v1.19 及更高版本 +- EKS：`docker` 适用于 v1.22 及以下版本；`containerd` 适用于 v1.23 及更高版本 +- GKE：`containerd` (请参阅 [Google 文档](https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd)了解更多信息) + + :::note + + 在安装过程中一次只能选择一个容器运行时引擎。 + + ::: + +**通过集群工具导航并安装 NeuVector Chart：** + +1. 单击 **☰ > 集群管理**。 +1. 在 Clusters 页面上，转到要部署 NeuVector 的集群，然后单击 **Explore**。 +1. 点击左侧导航栏底部的**集群工具**。 +1. 按照上面的步骤 4 相应地选择你的容器运行时，然后再次单击**安装**。 + +### 从 Rancher UI 访问 NeuVector + +1. 导航到安装了 NeuVector 的集群的 Cluster Explorer。在左侧导航栏中，单击 **NeuVector**。 +1. 单击外部链接以转到 NeuVector UI。选择链接后，用户必须接受`最终用户许可协议`才能访问 NeuVector UI。 + +### 从 Rancher UI 卸载 NeuVector + +**通过 Apps 卸载：** + +1. 单击 **☰ > 集群管理**。 +1. 在 **Apps** 下，点击 **Installed Apps**。 +1. 在 `cattle-neuvector-system` 下，选择 NeuVector 应用程序（如果需要，还可以选择相关的 CRD），然后单击**删除**。 + +**通过集群工具卸载：** + +1. 单击 **☰ > 集群管理**。 +1. 单击屏幕左下角的**集群工具**，然后单击 NeuVector Chart 下方的垃圾桶图标。如果需要，选择`删除与此应用关联的 CRD`，然后单击**删除**。 + +### GitHub 仓库 + +NeuVector 项目在[这里](https://github.com/neuvector/neuvector)。 + +### 文档 + +NeuVector 文档在[这里](https://open-docs.neuvector.com/)。 + +### 架构 + +NeuVector 安全解决方案包含四种类型的安全容器，分别是 Controller、Enforcer、Manager 和 Scanner。它还提供了一个称为 All-in-One 的特殊容器（主要用于 Docker 原生部署），能将 Controller、Enforcer 和 Manager 功能组合在一个容器中。此外，还有一个 Updater，运行该程序时会更新 CVE 数据库。 + +- **Controller：** 管理 NeuVector Enforcer 容器；为管理控制台提供 REST API。 +- **Enforcer：** 执行安全策略。 +- **Manager：** 提供一个 web-UI 和 CLI 控制台来管理 NeuVector 平台。 +- **All-in-One：** 包括 Controller、Enforcer 和 Manager。 +- **Scanner：** 对镜像、容器和节点执行漏洞和合规性扫描。 +- **Updater：** 更新 Neuvector 的 CVE 数据库（运行的时候）；重新部署 scanner pod。 + +
**NeuVector 安全容器：**
+ +![NeuVector 安全容器](/img/neuvector-security-containers.png) + +
**NeuVector 架构：**
+ +![NeuVector 架构](/img/neuvector-architecture.png) + +要了解有关 NeuVector 架构的更多信息，请参阅[此处](https://open-docs.neuvector.com/basics/overview#architecture)。 + +### CPU 和内存分配 + +以下是默认 NeuVector Chart 安装部署的最低计算资源推荐。请注意，未设置资源限制。 + +| 容器 | CPU - 请求 | 内存 - 请求 | +| ---------- | ----------------------------- | ----------- | +| Controller | 3（每个控制器需要 1GB 1vCPU） | \* | +| Enforcer | 所有节点上 (500MB .5vCPU) | 1GB | +| Manager | 1 (500MB .5vCPU) | \* | +| Scanner | 3 (100MB .5vCPU) | \* | + +\* Controller、Manager 和 Scanner 容器合计至少需要 1GB 内存。 + +### 强化集群支持 - Calico 和 Canal + + + + +- 如果 PSP 设置为 true，则所有 NeuVector 组件都是可部署的。 + +你需要为强化集群环境进行额外的配置，如下所示： + +1. 单击 **☰ > 集群管理**。 +1. 选择你创建的集群，并点击 **Explore**。 +1. 在左侧导航栏中，点击 **Apps**。 +1. 安装（或升级到）NeuVector 版本 `100.0.1+up2.2.2`。 + +- 在**编辑选项** > **其它配置**下，选中复选框来启用 **Pod 安全策略**（无需其他配置）： + + ![为 RKE1 强化集群启用 PSP](/img/psp-nv-rke.png) + +1. 点击右下角的**安装**。 + + + + +- 如果 PSP 设置为 true，则可以部署 NeuVector 组件 Controller 和 Enforcer。 + +**仅适用于 NeuVector Chart 版本 100.0.0+up2.2.0：** + +- 对于 Manager、Scanner 和 Updater 组件，需要进行额外的配置，如下所示： + +``` +kubectl patch deploy neuvector-manager-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}' +kubectl patch deploy neuvector-scanner-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}' +kubectl patch cronjob neuvector-updater-pod -n cattle-neuvector-system --patch '{"spec":{"jobTemplate":{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}}}' +``` + +
+ +你需要为强化集群环境进行额外的配置。 + +> **注意：** 你必须更新 RKE2 和 K3s 强化集群中的配置，如下所示。 + +1. 单击 **☰ > 集群管理**。 +1. 选择你创建的集群，并点击 **Explore**。 +1. 在左侧导航栏中，点击 **Apps**。 +1. 安装（或升级到）NeuVector 版本 `100.0.1+up2.2.2`。 + +- 在**编辑选项** > **其它配置**下，选中复选框来启用 **Pod 安全策略**。请注意，对于 `Manager runAsUser ID`，`Scanner runAsUser ID` 和 `Updater runAsUser ID`，你还必须输入大于 `0` 的值： + + ![为 RKE2 和 K3s 强化集群启用 PSP](/img/psp-nv-rke2.png) + +1. 点击右下角的**安装**。 + + + + +### 启用 SELinux 的集群支持 - Calico 和 Canal + +要在 RKE2 集群上启用 SELinux，请执行以下步骤： + +- 如果 PSP 设置为 true，则可以部署 NeuVector 组件 Controller 和 Enforcer。 + +**仅适用于 NeuVector Chart 版本 100.0.0+up2.2.0：** + +- 对于 Manager、Scanner 和 Updater 组件，需要进行额外的配置，如下所示： + +``` +kubectl patch deploy neuvector-manager-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}' +kubectl patch deploy neuvector-scanner-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}' +kubectl patch cronjob neuvector-updater-pod -n cattle-neuvector-system --patch '{"spec":{"jobTemplate":{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}}}' +``` + +### 离线环境中的集群支持 + +- 所有 NeuVector 组件都可部署在离线环境中的集群上，无需任何额外配置。 + +### 支持限制 + +- 目前仅支持管理员和集群所有者。 + +- 不支持 Fleet 多集群部署。 + +- Windows 集群不支持 NeuVector。 + +### 其他限制 + +- 目前，如果 NeuVector partner Chart 已存在，则 NeuVector 功能 Chart 的安装会失败。要解决此问题，请卸载 NeuVector partner Chart 并重新安装 NeuVector 功能 Chart。 + +- Controller 未准备好时，有可能无法从 Rancher UI 访问 NeuVector UI。在此期间，Controller 将尝试重新启动，并且需要几分钟才能进入 active 状态。 + +- 安装 NeuVector Chart 时，不会针对不同的集群类型自动检测容器运行时。要解决此问题，你可以手动指定运行时。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/rancher-desktop.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/rancher-desktop.md new file mode 100644 index 00000000000..5e418069595 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/rancher-desktop.md @@ -0,0 +1,33 @@ +--- +title: 桌面上的 Kubernetes 与 Rancher Desktop +--- + + + + + +Rancher Desktop 将用于桌面开发和测试云原生应用程序的基本工具捆绑在一起。 + +如果你在本地计算机上运行适用于云环境的应用程序，则通常需要进行大量准备工作。你需要选择一个容器运行时,安装 Kubernetes 和流行的实用程序，并且可能还需要设置虚拟机。单独安装组件并使它们协同工作可能是一个耗时的过程。 + +为了降低复杂性，Rancher Desktop 为团队提供了以下主要功能： + +- 在 macOS、Linux 和 Windows 操作系统上进行简单易用的安装。 +- K3s，一个易于使用的轻量级 Kubernetes 发行版。 +- 能够在 Kubernetes 版本之间轻松切换。 +- 由 Rancher 支持的基于 GUI 的集群仪表板，用于浏览本地集群。 +- 自由选择容器引擎：dockerd（moby）或 containerd。 +- 用于配置应用程序以满足你的需求的首选项设置。 +- 容器、基于 Kubernetes 的开发和操作工作流程所需的捆绑工具。 +- 定期更新，使捆绑工具保持最新。 +- 与流行的工具/IDE 集成，包括 VS Code 和 Skaffold。 +- 镜像和注册表访问控制。 +- 支持 Docker 扩展。 + +访问 [Rancher Desktop](https://rancherdesktop.io) 网站并阅读[文档](https://docs.rancherdesktop.io/)以了解更多信息。 + +要在你的机器上安装 Rancher Desktop，请参阅[安装指南](https://docs.rancherdesktop.io/getting-started/installation)。 + +## 在 Rancher Desktop 上尝试 Rancher + +Rancher Desktop 提供了轻松试用容器化、基于 Helm 的应用程序所需的设置和工具。你可以按照此[操作指南](https://docs.rancherdesktop.io/how-to-guides/rancher-on-rancher-desktop)，使用 Rancher Desktop 开始使用 Rancher Kubernetes 管理平台。 From 4f043afbb7f027756b8ae5643b662dff5321a681 Mon Sep 17 00:00:00 2001 From: jiandao <51182588+GGGitBoy@users.noreply.github.com> Date: Thu, 11 Apr 2024 05:19:03 +0800 Subject: [PATCH 44/67] =?UTF-8?q?Update=20missing=20Chinese=20translation?= =?UTF-8?q?=20about=20cis=E3=80=81istio=E3=80=81monitoring=20and=20logging?= =?UTF-8?q?=20parts=20for=20latest=20and=20v2.8=20=20(#1206)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Update missing translation about cis、istio、monitoring and logging parts for latest * Update missing translation about cis、istio、monitoring and logging parts for v2.8 --- .../cis-scan-guides/cis-scan-guides.md | 17 +++ .../istio-setup-guide/istio-setup-guide.md | 34 +++++ .../monitoring-alerting-guides.md | 15 ++ .../prometheus-federator-guides.md | 12 ++ .../advanced-configuration.md | 19 +++ .../advanced-configuration/prometheusrules.md | 2 +- .../monitoring-v2-configuration-guides.md | 55 +++++++ .../cis-scans/cis-scans.md | 121 +++++++++++++++ .../configuration-options.md | 43 ++++++ .../integrations-in-rancher/istio/istio.md | 140 ++++++++++++++++++ .../custom-resource-configuration.md | 12 ++ .../logging/logging.md | 115 ++++++++++++++ .../monitoring-and-alerting.md | 106 +++++++++++++ .../monitoring-v2-configuration.md | 15 ++ .../prometheus-federator.md | 109 ++++++++++++++ .../cis-scan-guides/cis-scan-guides.md | 17 +++ .../istio-setup-guide/istio-setup-guide.md | 34 +++++ .../monitoring-alerting-guides.md | 15 ++ .../prometheus-federator-guides.md | 12 ++ .../advanced-configuration.md | 19 +++ .../advanced-configuration/prometheusrules.md | 2 +- .../monitoring-v2-configuration-guides.md | 55 +++++++ .../cis-scans/cis-scans.md | 121 +++++++++++++++ .../configuration-options.md | 43 ++++++ .../integrations-in-rancher/istio/istio.md | 140 ++++++++++++++++++ .../custom-resource-configuration.md | 12 ++ .../logging/logging.md | 115 ++++++++++++++ .../monitoring-and-alerting.md | 106 +++++++++++++ .../monitoring-v2-configuration.md | 15 ++ .../prometheus-federator.md | 109 ++++++++++++++ 30 files changed, 1628 insertions(+), 2 deletions(-) create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-alerting-guides/monitoring-alerting-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/prometheus-federator-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/advanced-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cis-scans/cis-scans.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/istio/configuration-options/configuration-options.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/istio/istio.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/logging/custom-resource-configuration/custom-resource-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/logging/logging.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/prometheus-federator/prometheus-federator.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/monitoring-alerting-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/prometheus-federator-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/advanced-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cis-scans/cis-scans.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/istio/configuration-options/configuration-options.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/istio/istio.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/logging/custom-resource-configuration/custom-resource-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/logging/logging.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/prometheus-federator/prometheus-federator.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md new file mode 100644 index 00000000000..ff6b6200e19 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md @@ -0,0 +1,17 @@ +--- +title: CIS 扫描指南 +--- + + + + + +- [安装 Rancher CIS Benchmark](install-rancher-cis-benchmark.md) +- [卸载 Rancher CIS Benchmark](uninstall-rancher-cis-benchmark.md) +- [运行扫描](run-a-scan.md) +- [定时运行扫描](run-a-scan-periodically-on-a-schedule.md) +- [跳过测试](skip-tests.md) +- [查看报告](view-reports.md) +- [为 Rancher CIS Benchmark 启用告警](enable-alerting-for-rancher-cis-benchmark.md) +- [为定时扫描配置告警](configure-alerts-for-periodic-scan-on-a-schedule.md) +- [为集群扫描创建自定义 Benchmark 版本](create-a-custom-benchmark-version-to-run.md) \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md new file mode 100644 index 00000000000..e6b3e08e753 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md @@ -0,0 +1,34 @@ +--- +title: 设置指南 +--- + + + + + +本文介绍如何启用 Istio 并在你的项目中使用它。 + +如果你使用 Istio 进行流量管理，则需要允许外部流量进入集群。在这种情况下，你将需要执行以下所有步骤。 + +## 先决条件 + +本指南假设你已经[安装 Rancher](../../../getting-started/installation-and-upgrade/installation-and-upgrade.md)，且已经[配置了一个单独的 Kubernetes 集群](../../new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)并要在该集群上安装 Istio。 + +集群中的节点必须满足 [CPU 和内存要求](../../../integrations-in-rancher/istio/cpu-and-memory-allocations.md)。 + +Istio 控制的工作负载和服务必须满足 [Istio 要求](https://istio.io/docs/setup/additional-setup/requirements/)。 + +## 安装 + +:::tip 快速设置提示: + +如果你不需要外部流量到达 Istio，而只想设置 Istio 以监控和跟踪集群内的流量，请跳过[设置 Istio Gateway](set-up-istio-gateway.md)和[设置 Istio 的流量管理组件](set-up-traffic-management.md)步骤。 + +::: + +1. [在集群中启用 Istio。](enable-istio-in-cluster.md) +2. [在命名空间中启用 Istio。](enable-istio-in-namespace.md) +3. [使用 Istio Sidecar 添加部署和服务。](use-istio-sidecar.md) +4. [设置 Istio Gateway。](set-up-istio-gateway.md) +5. [设置 Istio 的流量管理组件。](set-up-traffic-management.md) +6. [生成和查看流量。](generate-and-view-traffic.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-alerting-guides/monitoring-alerting-guides.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-alerting-guides/monitoring-alerting-guides.md new file mode 100644 index 00000000000..16a504ae226 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-alerting-guides/monitoring-alerting-guides.md @@ -0,0 +1,15 @@ +--- +title: Monitoring/Alerting 指南 +--- + + + + + +- [启用 Monitoring](enable-monitoring.md) +- [卸载 Monitoring](uninstall-monitoring.md) +- [为工作负载设置 Monitoring](set-up-monitoring-for-workloads.md) +- [自定义 Grafana 仪表板](customize-grafana-dashboard.md) +- [持久化 Grafana 仪表板](create-persistent-grafana-dashboard.md) +- [调试高内存用量](debug-high-memory-usage.md) + diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/prometheus-federator-guides.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/prometheus-federator-guides.md new file mode 100644 index 00000000000..7bc98a902e4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/prometheus-federator-guides.md @@ -0,0 +1,12 @@ +--- +title: Prometheus Federator 指南 +--- + + + + + +- [启用 Prometheus Federator](enable-prometheus-federator.md) +- [卸载 Prometheus Federator](uninstall-prometheus-federator.md) +- [自定义 Grafana 仪表板](customize-grafana-dashboards.md) +- [为工作负载设置 Prometheus Federator](set-up-workloads.md) \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/advanced-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/advanced-configuration.md new file mode 100644 index 00000000000..549bc6fa852 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/advanced-configuration.md @@ -0,0 +1,19 @@ +--- +title: 高级配置 +--- + + + + + +### Alertmanager + +有关配置 Alertmanager 自定义资源的信息，请参阅[此页面。](alertmanager.md) + +### Prometheus + +有关配置 Prometheus 自定义资源的信息，请参阅[此页面。](prometheus.md) + +### PrometheusRules + +有关配置 PrometheusRules 自定义资源的信息，请参阅[此页面。](prometheusrules.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md index fc12de7e488..458011a703c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md @@ -1,5 +1,5 @@ --- -title: 配置 PrometheusRule +title: PrometheusRule 配置 --- PrometheusRule 定义了一组 Prometheus 告警和/或记录规则。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md new file mode 100644 index 00000000000..3b53fe7214f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md @@ -0,0 +1,55 @@ +--- +title: 配置 +--- + + + + + +本文介绍在 Rancher UI 中配置 Monitoring V2 的一些最重要选项。 + +有关为 Prometheus 配置自定义抓取目标和规则的信息，请参阅 [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) 的上游文档。Prometheus Operator [设计文档](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md)中解释了一些最重要的自定义资源。Prometheus Operator 文档还可以帮助你设置 RBAC、Thanos 或进行自定义配置。 + +## 设置资源限制和请求 + +安装 `rancher-monitoring` 时可以配置 Monitoring 应用的资源请求和限制。有关默认限制的更多信息，请参阅[此页面](../../../reference-guides/monitoring-v2-configuration/helm-chart-options.md#配置资源限制和请求)。 + +:::tip + +在空闲集群上，Monitoring 可能会占用很多 CPU 资源。要提高性能，请关闭 Prometheus Adapter。 + +::: + +## Prometheus 配置 + +通常不需要直接编辑 Prometheus 自定义资源。 + +相反，要让 Prometheus 抓取自定义指标，你只需创建一个新的 ServiceMonitor 或 PodMonitor 来将 Prometheus 配置为抓取其他指标。 + +### ServiceMonitor 和 PodMonitor 配置 + +有关详细信息，请参阅[此页面](../../../reference-guides/monitoring-v2-configuration/servicemonitors-and-podmonitors.md)。 + +### 高级 Prometheus 配置 + +有关直接编辑 Prometheus 自定义资源（对高级用例可能有帮助）的更多信息，请参阅[此页面](advanced-configuration/prometheus.md)。 + +## Alertmanager 配置 + +Alertmanager 自定义资源通常不需要直接编辑。在常见用例中，你可以通过更新路由和接收器来管理告警。 + +路由和接收器是 Alertmanager 自定义资源配置的一部分。在 Rancher UI 中，路由（Route）和接收器（Receiver）并不是真正的自定义资源，而是 Prometheus Operator 用来将你的配置与 Alertmanager 自定义资源同步的伪自定义资源。当路由和接收器更新时，Monitoring 应用将自动更新 Alertmanager 来反映这些更改。 + +对于一些高级用例，你可能需要直接配置 Alertmanager。有关详细信息，请参阅[此页面](advanced-configuration/alertmanager.md)。 + +### 接收器 + +接收器（Receiver）用于设置通知。有关如何配置接收器的详细信息，请参阅[此页面](../../../reference-guides/monitoring-v2-configuration/receivers.md)。 + +### 路由 + +路由（Route）在通知到达接收器之前过滤它们。每条路由都需要引用一个已经配置好的接收器。有关如何配置路由的详细信息，请参阅[此页面](../../../reference-guides/monitoring-v2-configuration/routes.md)。 + +### 高级配置 + +有关直接编辑 Alertmanager 自定义资源（对高级用例可能有帮助）的更多信息，请参阅[此页面](advanced-configuration/alertmanager.md)。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cis-scans/cis-scans.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cis-scans/cis-scans.md new file mode 100644 index 00000000000..41ca4d923cb --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cis-scans/cis-scans.md @@ -0,0 +1,121 @@ +--- +title: CIS 扫描 +--- + + + + + +Rancher 可以通过运行安全扫描来检查 Kubernetes 是否按照 CIS Kubernetes Benchmark 中定义的安全最佳实践进行部署。CIS 扫描可以运行在任何 Kubernetes 集群，包括托管的 Kubernetes，例如 EKS、AKS 和 GKE。 + +`rancher-cis-benchmark` 应用使用了 kube-bench, ，这是 Aqua Security 的开源工具，用于检查集群是否符合 CIS Kubernetes Benchmark。此外，为了生成集群级别的报告，此应用使用了 Sonobuoy 来聚合报告。 + +## 关于 CIS Benchmark + +CIS（Center for Internet Security）是一个 501(c\)(3) 非营利组织，成立于 2000 年 10 月，其使命是识别、开发、验证、促进和维持网络防御的最佳实践方案，并建立和指导社区，以在网络空间中营造信任的环境。该组织总部位于纽约东格林布什，其成员包括大公司、政府机构和学术机构。 + +CIS Benchmark 是目标系统安全配置的最佳实践。CIS Benchmark 是由安全专家、技术供应商、公开和私人社区成员，以及 CIS Benchmark 开发团队共同志愿开发的。 + +在 CIS 网站上[注册](https://learn.cisecurity.org/benchmarks)以查看官方 Benchmark 文档。 + +## 关于生成的报告 + +每次扫描都会生成一份报告，你可以在 Rancher UI 中查看该报告，并以 CSV 格式下载它。 + +默认情况下使用 CIS Benchmark v1.6。 + +Benchmark 版本包含在生成的报告中。 + +Benchmark 提供两种类型的建议，分别是自动（Automated）和手动（Manual）。Benchmark 中标记为 Manual 的建议不包含在生成的报告中。 + +一些测试会被标记为“不适用”。由于 Rancher 配置 RKE 集群的方式，这些测试不会在任何 CIS 扫描中运行。有关如何审核测试结果，以及为什么某些测试会被标记为不适用，请参阅 Rancher 的 Kubernetes 对应版本的[自测指南](../../reference-guides/rancher-security/rancher-security.md#CIS-Benchmark-和自我评估)。 + +该报告包含以下信息： + +| 报告中的列 | 描述 | +|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | CIS Benchmark 的 ID 号。 | +| `description` | CIS Benchmark 测试的描述。 | +| `remediation` | 为了通过测试需要修复的内容。 | +| `state` | 测试的状态，可以是通过、失败、跳过或不适用。 | +| `node_type` | 节点角色，角色决定了在节点上运行的测试。主测试在 controlplane 节点上运行，etcd 测试在 etcd 节点上运行，节点测试在 Worker 节点上运行。 | +| `audit` | 这是 `kube-bench` 为此测试运行的审计检查。 | +| `audit_config` | 适用于审计脚本的任何配置。 | +| `test_info` | `kube-bench` 报告的测试相关信息（如果存在）。 | +| `commands` | `kube-bench` 报告的测试相关的命令（如果存在）。 | +| `config_commands` | `kube-bench` 报告的测试相关的配置数据（如果存在）。 | +| `actual_value` | 测试的实际值。如果由 `kube-bench` 报告，则会显示。 | +| `expected_result` | 测试的预期值。如果由 `kube-bench` 报告，则会显示。 | + +请参阅[集群加固指南中的表格](../../reference-guides/rancher-security/rancher-security.md)，以了解 Kubernetes、Benchmark、Rancher 以及我们的集群强化指南的版本对应关系。另外，请参阅强化指南，以获取符合 CIS 的集群的配置文件以及修复失败测试的信息。 + +## 测试配置文件 + +以下是可用的配置文件： + +- Generic CIS 1.6 +- Generic CIS 1.20 +- Generic CIS 1.23 +- RKE permissive 1.6 +- RKE hardened 1.6 +- RKE permissive 1.20 +- RKE hardened 1.20 +- RKE permissive 1.23 +- RKE hardened 1.23 +- RKE2 permissive 1.6 +- RKE2 hardened 1.6 +- RKE2 permissive 1.20 +- RKE2 hardened 1.20 +- RKE2 permissive 1.23 +- RKE2 hardened 1.23 +- K3s permissive 1.6 +- K3s hardened 1.6 +- K3s permissive 1.20 +- K3s hardened 1.20 +- K3s permissive 1.23 +- K3s hardened 1.23 +- AKS +- EKS +- GKE + +你还可以通过保存一组要跳过的测试来自定义配置文件。 + +所有配置文件都会有一组不适用的测试，CIS 扫描会跳过这些测试。RKE 集群管理 Kubernetes 的方式导致这些测试被认为不适用。 + +RKE 集群扫描配置文件有两种类型： + +- **Permissive**：此配置文件有一组要跳过的测试，跳过的原因是这些测试会在默认的 RKE Kubernetes 集群上失败。除了跳过的测试列表之外，配置文件也不会运行不适用的测试。 +- **Hardened**：此配置文件不会跳过任何测试（不适用的测试除外）。 + +EKS 和 GKE 集群扫描的配置文件基于这些集群类型特定的 CIS Benchmark 版本。 + +要通过 “Hardened” 配置文件，你需要遵从[强化指南](../../reference-guides/rancher-security/rancher-security.md#Rancher-加固指南)并使用强化指南中定义的 `cluster.yml` 来配置一个强化集群。 + +默认配置文件和支持的 CIS Benchmark 版本取决于扫描的集群类型： + +`rancher-cis-benchmark` 支持 CIS 1.6 Benchmark 版本。 + +- RKE Kubernetes 集群默认使用 RKE Permissive 1.6 配置文件。 +- EKS 和 GKE 有自己的 CIS Benchmark，由 `kube-bench` 发布。这些集群默认使用相应的测试配置文件。 +- RKE2 Kubernetes 集群默认使用 RKE2 Permissive 1.6 配置文件。 +- RKE、RKE2、EKS 和 GKE 以外的集群类型默认使用 Generic CIS 1.5 配置文件。 + +## 跳过和不适用的测试 + +有关要跳过和不适用的测试列表，请参阅[此页面](../../how-to-guides/advanced-user-guides/cis-scan-guides/skip-tests.md)。 + +目前，只有用户定义的跳过测试会在生成报告中标记为跳过。 + +如果某个默认配置文件将某个测试定义为跳过，则该测试也会标记为不适用。 + +## RBAC + +有关权限的详细信息，请参阅[此页面](rbac-for-cis-scans.md)。 + +## 配置 + +有关为扫描、配置文件和 Benchmark 版本配置自定义资源的更多信息，请参阅[此页面](configuration-reference.md)。 + +## 操作指南 + +要了解如何运行 CIS 扫描，请参阅 [CIS 扫描指南](../../how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/istio/configuration-options/configuration-options.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/istio/configuration-options/configuration-options.md new file mode 100644 index 00000000000..800b4e576da --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/istio/configuration-options/configuration-options.md @@ -0,0 +1,43 @@ +--- +title: 配置选项 +--- + + + + + +### Egress 支持 + +默认情况下，Egress 网关是禁用的，但你可以在安装或升级时使用 values.yaml 或[覆盖文件](#覆盖文件)启用它。 + +### 启用自动 Sidecar 注入 + +默认情况下，自动 sidecar 注入是禁用的。要启用此功能，请在安装或升级时在 values.yaml 中设置 `sidecarInjectorWebhook.enableNamespacesByDefault=true`。这会自动将 Istio sidecar 注入到所有已部署的新命名空间。 + +### 覆盖文件 + +覆盖文件用于为 Istio 进行更广泛的配置。它允许你更改 [IstioOperator API](https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/) 中可用的任何值。你可以自定义默认安装以满足你的需求。 + +覆盖文件将在 Istio Chart 默认安装的基础上添加配置。换言之，你不需要为安装中已定义的组件进行重新定义。 + +有关覆盖文件的更多信息，请参阅 [Istio 文档](https://istio.io/latest/docs/setup/install/istioctl/#configure-component-settings) + +### 选择器和抓取配置 + +Monitoring 应用设置了 `prometheus.prometheusSpec.ignoreNamespaceSelectors=false`，即在默认情况下跨所有命名空间进行监控。这样，你可以查看部署在具有 `istio-injection=enabled` 标签的命名空间中的资源的流量、指标和图。 + +如果你想将 Prometheus 限制为特定的命名空间，请设置 `prometheus.prometheusSpec.ignoreNamespaceSelectors=true`。完成此操作后，你需要添加其他配置来继续监控你的资源。 + +详情请参阅[本节](selectors-and-scrape-configurations.md)。 + +### 在具有 Pod 安全策略的情况下启用 Istio + +详情请参阅[本节](pod-security-policies.md)。 + +### 在 RKE2 集群上安装 Istio 的其他步骤 + +详情请参阅[本节](install-istio-on-rke2-cluster.md)。 + +### 项目网络隔离的其他步骤 + +详情请参阅[本节](project-network-isolation.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/istio/istio.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/istio/istio.md new file mode 100644 index 00000000000..42c642b4b64 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/istio/istio.md @@ -0,0 +1,140 @@ +--- +title: Istio +--- + + + + + +[Istio](https://istio.io/) 是一种开源工具，可以让 DevOps 团队更轻松地观察、控制、排查并保护复杂的微服务网络中的流量。 + +随着微服务网络的变化和增长，微服务网络之间的交互变得越来越难以管理和理解。在这种情况下，将服务网格作为单独的基础设施层是非常有用的。Istio 的服务网格可以让你在不直接更改微服务的情况下控制微服务之间的流量。 + +Rancher 与 Istio 集成，使得管理员或集群所有者可以将 Istio 交给开发者团队，然后开发者使用 Istio 执行安全策略，排查问题，或为蓝绿部署，金丝雀部署，和 A/B 测试进行流量管理。 + +此核心服务网格支持但不限于以下功能： + +- **管理流量**：例如入口和出口路由、断路、镜像。 +- **安全**：具有用于验证和授权流量和用户的资源，包括 mTLS。 +- **可观察性**：观察日志、指标和分布式流量。 + +[设置 Istio](../../how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md) 后，你可以通过 Rancher UI、`kubectl` 或 ` Istioctl` 来使用 Istio 的 controlplane 功能。 + +Istio 需要由 `cluster-admin` 设置后才能在项目中使用。 + +## Rancher 2.5 的新功能 + +Istio 已简化了整体架构。结合 Pilot、Citadel、Galley 和 sidecar injector 创建了一个单独的组件 Istiod。Node Agent 功能也已合并到 istio-agent 中。 + +以前由 Istio 安装的插件（cert-manager、Grafana、Jaeger、Kiali、Prometheus、Zipkin）现在需要单独安装。Istio 支持安装来自 Istio 项目的集成，并保持与非 Istio 项目的兼容性。 + +你仍然可以通过安装 [Rancher Monitoring](../monitoring-and-alerting/monitoring-and-alerting.md) 或安装你自己的 Prometheus operator 来使用 Prometheus 集成。Rancher 的 Istio chart 还默认安装 Kiali，确保你可以开箱即用地全面了解微服务。 + +Istio 已经脱离了使用 Helm 安装的方式，现在通过 Istioctl 二进制文件或 Istio Operator 进行安装。为了使用最简单的方式与 Istio 交互，Rancher 的 Istio 会维护一个 Helm Chart，该 Chart 使用 Istioctl 二进制文件来管理你的 Istio 安装。 + +此 Helm Chart 将在 UI 的**应用 & 市场**中提供。有权访问 Rancher Chart 应用商店的用户需要先设置 Istio，然后才能在项目中使用它。 + +## Istio 附带的工具 + +我们的 [Istio](https://istio.io/) 安装程序将 istioctl 二进制命令包装在一个 Helm chart 中，其中包括一个覆盖文件的选项，用来支持复杂的自定义配置。 + +它还包括以下内容： + +### Kiali + +Kiali 是一个全面的可视化辅助工具，用于绘制整个服务网格中的流量图。它允许你查看它们的连接方式，包括它们之间的流量速率和延迟。 + +你可以检查服务网格的运行状况，或深入查看单个组件的传入和传出请求。 + +### Jaeger + +Jaeger 是用于跟踪分布式系统的工具。我们的 Istio 安装程序包括能快速启动的一体化 [Jaeger](https://www.jaegertracing.io/) 安装。 + +请注意，这不是符合 Jaeger 生产要求的部署。此部署使用在内存中的存储组件，而 Jaeger 推荐在生产环境中使用持久存储组件。有关你所需的部署策略的更多信息，请参阅 [Jaeger 文档](https://www.jaegertracing.io/docs/latest/operator/#production-strategy)。 + +## 先决条件 + +在启用 Istio 之前，建议你先确认你的 Rancher worker 节点是否有足够的 [CPU 和内存](cpu-and-memory-allocations.md)来运行 Istio 的所有组件。 + +如果要在 RKE2 集群上安装 Istio，则需要执行一些额外的步骤。有关详细信息，请参阅[本节](#在-rke2-集群上安装-istio-的其他步骤) + +## 设置指南 + +如需了解如何设置 Istio 并在项目中使用它，请参阅[设置指南](../../how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md)。 + +## 卸载 Istio + +要从集群、命名空间或工作负载中删除 Istio 组件，请参阅[卸载 Istio](disable-istio.md)。 + +## 访问可视化 + +> 默认情况下，只有 cluster-admin 可以访问 Kiali。有关如何允许具有管理员、编辑或查看权限的角色访问它们的说明，请参阅[本节](rbac-for-istio.md)。 + +在集群中设置 Istio 后，你可以在 Rancher UI 中使用 Grafana、Prometheus 和 Kiali。 + +要访问 Grafana 和 Prometheus 可视化： + +1. 在左上角，单击 **☰ > 集群管理**。 +2. 在**集群**页面上，转到要可视化的集群，然后单击 **Explore**。 +3. 在左侧导航栏中，单击**监控**。 +4. 点击 **Grafana** 或任何其他仪表板。 + +要访问 Kiali 可视化： + +1. 在左上角，单击 **☰ > 集群管理**。 +2. 在**集群**页面上，转到要查看 Kiali 的集群，然后单击 **Explore**。 +3. 在左侧导航栏中，单击 **Istio**。 +4. 单击 **Kiali**。从这里，你可以访问**流量图**或**流量指标**选项卡，从而可视化网络指标。 + +默认情况下，prometheus 会拾取所有命名空间，并将数据用于 Kiali 图。如果你想使用不同的配置进行 prometheus 数据抓取，请参阅[选择器/抓取配置](configuration-options/selectors-and-scrape-configurations.md)。 + +你的角色决定了你对可视化的访问。只有 `cluster-admin` 角色可以使用 Grafana 和 Prometheus。默认情况下，只有 `cluster-admin` 可以使用 Kiali UI，但是 `cluster-admin` 可以通过编辑 Istio values.yaml 来允许其他角色进行访问。 + +## 架构 + +Istio 安装了一个服务网格，它使用 [Envoy](https://www.envoyproxy.io) Sidecar 代理来拦截到每个工作负载的流量。这些 sidecar 拦截并管理服务之间的通信，从而实现精细化观察并控制集群内的流量。 + +只有注入了 Istio sidecar 的工作负载可以通过 Istio 进行跟踪和控制。 + +如果命名空间启用了 Istio，部署到命名空间的新工作负载会自动具有 Istio sidecar。你需要为之前的工作负载手动启用 Istio。 + +有关 Istio sidecar 的更多信息，请参阅 [Istio sidecare-injection 文档](https://istio.io/docs/setup/kubernetes/additional-setup/sidecar-injection/)。有关 Istio 架构的更多信息，请参阅 [Istio 架构文档](https://istio.io/latest/docs/ops/deployment/architecture/)。 + +### 多个 Ingress + +默认情况下，每个 Rancher 配置的集群都有一个 NGINX Ingress Controller 来允许流量进入集群。Istio 还在 `istio-system` 命名空间中默认安装一个 Ingress Gateway。因此，你的集群将有两个 ingress。 + +![启用 Istio 的集群可以有两个 ingress，分别是默认的 Nginx ingress 和默认的 Istio controller](/img/istio-ingress.svg) + +可以通过[覆盖文件](configuration-options/configuration-options.md#覆盖文件)来启用其他 Istio Ingress Gateway。 + +### Egress 支持 + +默认情况下，Egress 网关是禁用的，但你可以在安装或升级时使用 values.yaml 或[覆盖文件](configuration-options/configuration-options.md#覆盖文件)启用它。 + +## 在 RKE2 集群上安装 Istio 的其他步骤 + +要在 RKE2 集群上安装 Istio，请按照[步骤](configuration-options/install-istio-on-rke2-cluster.md)进行操作。 + +## 在离线环境中升级 Istio + +现在，Istio Pod 安全策略默认启用。新值 `installer.releaseMirror.enabled` 已添加到 rancher-istio Chart 中，以启用和禁用支持离线升级的 Server。请注意，`installer.releaseMirror.enabled` 默认设置为 `false`。你可以在安装或升级时根据需要设置该值。按照以下步骤执行： + +1. 在 Rancher UI 中配置离线 Rancher 实例和离线自定义集群。 +2. 在集群中安装 Monitoring：**Cluster Explorer > Apps & Marketplace > Charts > Monitoring**。 +3. 将 Istio 所需的所有镜像拉入在离线环境中使用的私有镜像仓库。 +4. 在集群中安装 Istio：**Cluster Explorer > Apps & Marketplace > Charts > Istio**。 + +:::note + +你可以在新安装的 Istio 上启用 [Jaeger](https://www.jaegertracing.io/) 和 [Kiali](https://kiali.io/)。为确保 Jaeger 和 Kiali 正常工作，请在安装期间将 `values.yaml` 中的 `installer.releaseMirror.enabled` 设置为 `true`。 + +::: + +5. 升级 Istio。 + +:::caution + +如果你还没有执行操作，请设置 `installer.releaseMirror.enabled=true` 以升级 Istio。 + +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/logging/custom-resource-configuration/custom-resource-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/logging/custom-resource-configuration/custom-resource-configuration.md new file mode 100644 index 00000000000..d719b72fe1d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/logging/custom-resource-configuration/custom-resource-configuration.md @@ -0,0 +1,12 @@ +--- +title: 自定义资源配置 +--- + + + + + +通过以下自定义资源配置 logging： + +- [Flow 和 ClusterFlow](flows-and-clusterflows.md) +- [Output 和 ClusterOutput](outputs-and-clusteroutputs.md) \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/logging/logging.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/logging/logging.md new file mode 100644 index 00000000000..dcc65014fbd --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/logging/logging.md @@ -0,0 +1,115 @@ +--- +title: Rancher Logging 集成 +description: Rancher 集成了主流的日志服务。了解集成日志服务的要求和优势，并在你的集群上启用 Logging。 +--- + + + + + +现在，Rancher 的日志管理由 [Logging operator](https://kube-logging.github.io/docs/) 提供支持，它取代了以前的内部解决方案。 + +## 启用 Logging + +你可以转到**应用**页面并安装 Logging 应用程序，从而为 Rancher 管理的集群启用 Logging： + +1. 转到要安装 Logging 的集群，然后单击 **Apps**。 +1. 点击 **Logging** 应用。 +1. 滚动到 Helm Chart README 的底部，然后单击**安装**。 + +**结果**：Logging 应用已部署到 `cattle-logging-system` 命名空间中。 + +## 卸载 Logging + +1. 转到要安装 Logging 的集群，然后单击 **Apps**。 +1. 点击**已安装的应用**。 +1. 转到 `cattle-logging-system` 命名空间并选中 `rancher-logging` 和 `rancher-logging-crd` 框。 +1. 单击**删除**。 +1. 确认**删除**。 + +**结果**：已卸载 `rancher-logging`。 + +## 架构 + +有关 Logging 应用程序工作原理的更多信息，请参阅[本节](logging-architecture.md)。 + +## RBAC + +Rancher Logging 有两个角色，分别是 `logging-admin` 和 `logging-view`。有关如何以及何时使用这些角色的更多信息，请参阅[此页面](rbac-for-logging.md)。 + +## 配置 Logging 自定义资源 + +要管理 `Flows`、`ClusterFlows`、`Outputs` 和 `ClusterOutputs`： + +1. 在左上角，单击 **☰ > 集群管理**。 +2. 在**集群**页面上，转到要配置 Logging 自定义资源的集群，然后单击 **Explore**。 +3. 在左侧导航栏中，单击 **Logging**。 + +### Flows 和 ClusterFlows + +有关配置 `Flows` 和 `ClusterFlows` 的帮助，请参阅[此页面](custom-resource-configuration/flows-and-clusterflows.md)。 + +### Outputs 和 ClusterOutputs + +有关配置 `Outputs` 和 `ClusterOutputs` 的帮助，请参阅[此页面](custom-resource-configuration/outputs-and-clusteroutputs.md)。 + +## 配置 Logging Helm Chart + +有关在安装或升级 Logging 应用程序时可配置的选项，请参阅[此页面](logging-helm-chart-options.md)。 + +### Windows 支持 + +你可以从 Windows 节点[启用 Logging](logging-helm-chart-options.md#启用禁用-windows-节点-logging)。 + +### 使用自定义 Docker 根目录 + +有关使用自定义 Docker 根目录的详细信息，请参阅[本节](logging-helm-chart-options.md#使用自定义-docker-根目录)。 + +### 处理污点和容忍度 + +有关如何在 Logging 应用程序中使用污点和容忍度的信息，请参阅[此页面](taints-and-tolerations.md)。 + +### 在 SELinux 上使用 Logging V2 + +有关在启用了 SELinux 的节点上使用 Logging 应用程序的信息，请参阅[本节](logging-helm-chart-options.md#启用-logging-应用程序以使用-selinux)。 + +### 其他日志来源 + +默认情况下，Rancher 会收集所有类型集群的 controlplane 组件和节点组件的日志。在某些情况下，也会收集其他日志。有关详细信息，请参阅[本节](logging-helm-chart-options.md#其他日志来源)。 + +## 故障排除 + +### 日志缓冲区导致 Pod 过载 + +根据你的配置，默认缓冲区大小可能太大并导致 Pod 故障。减少负载的一种方法是降低记录器的刷新间隔。这可以防止日志溢出缓冲区。你还可以添加更多刷新线程来处理大量日志试图同时填充缓冲区的情况。 + +有关如何配置日志缓冲区来满足企业需求的更完整说明，请参阅[缓冲区](https://kube-logging.github.io/docs/configuration/plugins/outputs/buffer/)和 [Fluentd 配置](https://kube-logging.github.io/docs/logging-infrastructure/fluentd/)的官方 Logging Operator 文档。 + +### `cattle-logging` 命名空间正在重新创建 + +如果你的集群之前在旧版 Rancher UI 的全局视图中部署了 Logging，`cattle-logging` 命名空间可能会不断被重新创建。 + +要解决这个问题，你可以将所有 `clusterloggings.management.cattle.io` 和 `projectloggings.management.cattle.io` 自定义资源从管理集群中针对该集群的命名空间中删除。 +这些自定义资源会导致 Rancher 在下游集群中创建 `cattle-logging` 命名空间（如果不存在）。 + +集群命名空间与集群 ID 匹配，因此我们需要找到每个集群的集群 ID。 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面上，转到要获取 ID 的集群，然后单击 **Explore**。 +2. 从以下其中一个 URL 中复制 `` 的内容。`` 是集群命名空间名称。 + +```bash +# Cluster Management UI +https:///c// + +# Cluster Dashboard +https:///dashboard/c// +``` + +现在我们有了 `` 命名空间，我们可以删除导致 `cattle-logging` 不断重新创建的自定义资源。 +*警告*：请确保当前未使用 Logging （从旧版 Rancher UI 全局视图中安装的版本）。 + +```bash +kubectl delete crd clusterloggings.management.cattle.io -n +kubectl delete crd projectloggings.management.cattle.io -n +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md new file mode 100644 index 00000000000..cebb2e4323f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md @@ -0,0 +1,106 @@ +--- +title: 监控和告警 +description: Prometheus 允许你查看来自不同 Rancher 和 Kubernetes 对象的指标。了解监控范围以及如何启用集群监控 +--- + + + + + +你可以使用 `rancher-monitoring` 应用，将业界领先的开源监控和告警解决方案快速部署到你的集群中。 + +在 Rancher v2.5 中引入的 `rancher-monitoring` operator 由 [Prometheus](https://prometheus.io/)、[Grafana](https://grafana.com/grafana/)、[Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)， [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) 和 [Prometheus adapter](https://github.com/DirectXMan12/k8s-prometheus-adapter) 提供支持。 + +有关在 Rancher v2.2 到 v2.4 中可用的 V1 监控和告警的信息，请参阅 Rancher v2.0 到 v2.4 文档中的[集群监控](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/cluster-monitoring/cluster-monitoring.md)，[告警](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/cluster-alerts/cluster-alerts.md)，[通知](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/notifiers.md)和[工具](/versioned_docs/version-2.0-2.4/reference-guides/rancher-project-tools/rancher-project-tools.md)。 + +使用 `rancher-monitoring` 应用程序，你可以快速部署领先的开源监控和告警解决方案到你的集群上。 + +### 功能 + +Prometheus 支持查看 Rancher 和 Kubernetes 对象的指标。通过使用时间戳，Prometheus 能让你通过 Rancher UI 或 Grafana（与 Prometheus 一起部署的分析查看平台）以更容易阅读的图表和视觉形式来查询和查看这些指标。 + +通过查看 Prometheus 从集群的 controlplane、节点和 deployment 中抓取的数据，你可以随时了解集群中发生的所有事件。然后，你可以使用这些分析来更好地运行你的环境，例如在系统紧急情况发生之前阻止它们、制定维护策略，或恢复崩溃的服务器。 + +Monitoring 应用： + +- 监控集群节点、Kubernetes 组件和软件部署的状态和进程。 +- 根据 Prometheus 收集的指标定义告警。 +- 创建自定义 Grafana 仪表板。 +- 使用 Prometheus Alertmanager 通过电子邮件、Slack、PagerDuty 等配置告警通知。 +- 根据 Prometheus 收集的指标，将预先计算的、经常需要的，或计算成本高的表达式定义为新的时间序列。 +- 通过 Prometheus Adapter，将从 Prometheus 收集的指标公开给 Kubernetes Custom Metrics API，以便在 HPA 中使用。 + +有关监控组件如何协同工作的说明，请参阅 [Monitoring 工作原理](how-monitoring-works.md)。 + +## 默认组件和部署 + +### 内置仪表板 + +默认情况下，监控应用将 Grafana 仪表板（由 [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) 项目策划）部署到集群上。 + +它还部署一个 Alertmanager UI 和一个 Prometheus UI。有关这些工具的更多信息，请参见[内置仪表板](built-in-dashboards.md)。 + +### 默认指标 Exporter + +默认情况下，Rancher Monitoring 会部署 Exporter（例如 [node-exporter](https://github.com/prometheus/node_exporter) 和 [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)）。 + +这些默认 Exporter 会自动从 Kubernetes 集群的所有组件（包括工作负载）中抓取 CPU 和内存的指标。 + +### 默认告警 + +Monitoring 应用会默认部署一些告警。要查看默认告警，请转到 [Alertmanager UI](built-in-dashboards.md#alertmanager-ui) 并单击**展开所有组**。 + +### Rancher UI 中公开的组件 + +有关 Rancher UI 中公开的监控组件列表，以及编辑它们的常见用例，请参阅[本节](how-monitoring-works.md#rancher-ui-中公开的组件)。 + +## RBAC + +有关配置 monitoring 访问权限的信息，请参阅[此页面](rbac-for-monitoring.md)。 + +:::note + +Rancher 和 Project 的读取权限并不一定适用于监控相关资源. 查看 [monitoring-ui-view](rbac-for-monitoring.md#其他监控角色) 获取更多详细信息. + +::: + +## 指南 + +- [启用 monitoring](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md) +- [卸载 monitoring](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/uninstall-monitoring.md) +- [Monitoring 工作负载](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/set-up-monitoring-for-workloads.md) +- [自定义 Grafana 仪表板](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/customize-grafana-dashboard.md) +- [持久化 Grafana 仪表板](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/create-persistent-grafana-dashboard.md) +- [调试高内存使用率](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/debug-high-memory-usage.md) + +## 配置 + +### 在 Rancher 中配置 Monitoring 资源 + +此处的配置参考假设你已经熟悉 monitoring 组件的协同工作方式。如需更多信息，请参阅 [monitoring 的工作原理](how-monitoring-works.md)。 + +- [ServiceMonitor 和 PodMonitor](../../reference-guides/monitoring-v2-configuration/servicemonitors-and-podmonitors.md) +- [接收器](../../reference-guides/monitoring-v2-configuration/receivers.md) +- [路由](../../reference-guides/monitoring-v2-configuration/routes.md) +- [PrometheusRule](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md) +- [Prometheus](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheus.md) +- [Alertmanager](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/alertmanager.md) + +### 配置 Helm Chart 选项 + +有关 `rancher-monitoring` Chart 选项的更多信息，包括设置资源限制和请求的选项，请参阅 [Helm Chart 选项](../../reference-guides/monitoring-v2-configuration/helm-chart-options.md)。 + +## Windows 集群支持 + +如果 Monitoring 部署到 RKE1 Windows 集群，Monitoring V2 将自动部署 [windows-exporter](https://github.com/prometheus-community/windows_exporter) DaemonSet 并设置 ServiceMonitor，以从每个部署的 Pod 中收集指标。这将使用 `windows_` 指标填充 Prometheus，这些指标与 [node_exporter](https://github.com/prometheus/node_exporter) 为 Linux 主机导出的 `node_` 指标类似。 + +为了能够为 Windows 完全部署 Monitoring V2，你的所有 Windows 主机都必须至少具有 v0.1.0 的 [wins](https://github.com/rancher/wins) 版本。 + +有关如何在现有 Windows 主机上升级 wins 版本的更多信息，请参阅 [Windows 集群对 Monitoring V2 的支持](windows-support.md)。 + + +## 已知问题 + +有一个[已知问题](https://github.com/rancher/rancher/issues/28787#issuecomment-693611821)，即 K3s 集群需要的内存超过分配的默认内存。如果你在 K3s 集群上启用 Monitoring，将 `prometheus.prometheusSpec.resources.memory.limit` 设置为 2500 Mi，并将 `prometheus.prometheusSpec.resources.memory.request` 设置为 1750 Mi。 + +如需获取意见和建议，请参阅[调试高内存使用情况](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/debug-high-memory-usage.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md new file mode 100644 index 00000000000..1110b88e03a --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md @@ -0,0 +1,15 @@ +--- +title: Monitoring V2 配置 +--- + + + + + +以下将说明在 Rancher 中如何配置 Monitoring V2 的基本重要选项： + +- [接收器配置](receivers.md) +- [路由配置](routes.md) +- [ServiceMonitor 和 PodMonitor 配置](servicemonitors-and-podmonitors.md) +- [Helm Chart 选项](helm-chart-options.md) +- [示例](examples.md) \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/prometheus-federator/prometheus-federator.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/prometheus-federator/prometheus-federator.md new file mode 100644 index 00000000000..6e4c4bd8ac3 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/prometheus-federator/prometheus-federator.md @@ -0,0 +1,109 @@ +--- +title: Prometheus Federator +--- + + + + + +Prometheus Federator（也称为 Project Monitoring V2）基于 [rancher/helm-project-operator](https://github.com/rancher/helm-project-operator) 部署一个 Helm Project Operator。该 Operator 管理 Helm Chart 的部署，每个 Operator 都包含一个 Project Monitoring Stack，而每个堆栈都包含： + +- [Prometheus](https://prometheus.io/)（由 [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) 在外部管理） +- [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)（由 [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) 在外部管理） +- [Grafana](https://github.com/helm/charts/tree/master/stable/grafana)（通过嵌入式 Helm Chart 部署） +- 基于 [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus/) 社区策划资源集合的默认 PrometheusRules 和 Grafana 仪表板 +- 监视已部署资源的默认 ServiceMonitor + +:::note 重要提示： + +Prometheus Federator 适合在已安装 Prometheus Operator CRD 的集群中与现有的 Prometheus Operator Deployment 一起部署。 + +::: + +## Operator 工作原理 + +1. 在部署此 Chart 时，用户可以创建 ProjectHelmCharts CR，并在**项目 Registration 命名空间 (`cattle-project-`)** 中将 `spec.helmApiVersion` 设置为 `monitoring.cattle.io/v1alpha1`（在 Rancher UI 中也称为“项目监控”）。 +2. 在看到每个 ProjectHelmChartCR 时，Operator 会代表项目所有者在**项目 Release 命名空间 (`cattle-project--monitoring`)** 中自动部署一个 Project Prometheus 堆栈（基于 ProjectHelmChart 控制器在 **Operator / System Namespace** 中创建的 HelmChart CR 和 HelmRelease CR）。 +3. RBAC 将自动分配到项目 Release 命名空间中，从而允许用户查看 Prometheus、Alertmanager 以及已部署的 Project Monitoring Stack 的 Grafana UI（基于在项目 Registration 命名空间上针对[面向用户的默认 Kubernetes 角色](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles)定义的 RBAC）。有关详细信息，请参阅[配置 RBAC](rbac.md)。 + +### 什么是项目？ + +在 Prometheus Federator 中，项目是一组可以由 `metav1.LabelSelector` 标识的命名空间。默认情况下，用于标识项目的标签是 `field.cattle.io/projectId`，该标签用于标识给定 Rancher 项目中包含的命名空间。 + +### 配置由 ProjectHelmChart 创建的 Helm 版本 + +此 ProjectHelmChart 资源的 `spec.values` 对应于为底层 Helm Chart 配置的 `values.yaml` 覆盖，该 Helm Chart 是 Operator 代表用户部署的。要查看底层 Chart 的 `values.yaml` 规范，你可以选择以下其中一种方式： + +- 查看位于 [`charts/rancher-project-monitoring` 中的 `rancher/prometheus-federator`](https://github.com/rancher/prometheus-federator/blob/main/charts/rancher-project-monitoring) 的 Chart 定义（Chart 版本会绑定到 Operator 版本）。 +- 查找在每个项目 Registration 命名空间中自动创建的名为 `monitoring.cattle.io.v1alpha1` 的 ConfigMap，其中包含用于配置 Chart（直接嵌入到 `prometheus-federator` 二进制文件中）的`values.yaml` 和 `questions.yaml`。 + +### 命名空间 + +Prometheus Federator 是基于 [rancher/helm-project-operator](https://github.com/rancher/helm-project-operator) 的 Project Operator，Prometheus Federator 提供了三类命名空间供 Operator 查找： + +1. **Operator / System 命名空间**：部署 Operator 的命名空间（例如 `cattle-monitoring-system`）。此命名空间将包含该 Operator 监视的所有 ProjectHelmChart 的所有 HelmChart 和 HelmRelease。**只有集群管理员才能访问此命名空间**。 + +2. **项目 Registration 命名空间 (`cattle-project-`)**：Operator 在这些命名空间中监视 ProjectHelmChart。对于在项目发布命名空​​间中创建的自动分配的 RBAC，应用于此命名空间的 RoleBinding 和 ClusterRoleBinding 也会作为 RBAC 的真实来源。有关详细信息，请参阅 [RBAC 页面](rbac.md)。**项目所有者（admin）、项目成员（edit）和只读成员（view）应该有权访问此命名空间。** + + :::note 注意事项： + + - 如果提供了 `.Values.global.cattle.projectLabel`（默认设置为 `field.cattle.io/projectId`），则 Operator 会自动生成项目注册命名空间，并将它导入到命名空间绑定的项目中。换言之，如果观察到至少一个带有该标签的命名空间，则 Operator 会创建一个项目 Registration 命名空间。除非出现以下两种情况，否则 Operator 不会让这些命名空间被删除。第一种情况是带有该标签的所有命名空间都消失了（例如，这是该项目中的最后一个命名空间，在这种情况下，命名空间将标有标签 `"helm.cattle.io/helm-project-operator-orphaned": "true"`，表示可以删除）。第二种情况是由于项目 ID 是在 `.Values.helmProjectOperator.otherSystemProjectLabelValues` 下提供的（用作项目的拒绝名单），导致 Operator 不再监视该项目。这些命名空间不会被自动删除，这样能避免破坏用户数据。如果需要，建议用户在创建或删除项目时手动清理这些命名空间。 + + - 如果未提供 `.Values.global.cattle.projectLabel`，则 Operator / System 命名空间也是项目注册命名空间。 + + ::: + +3. **项目发布命名空​​间（`cattle-project--monitoring`）**：Operator 代表 ProjectHelmChart 在其中部署项目监控堆栈的命名空间集。Operator 还将根据在项目 Registration 命名空间中找到的绑定，自动为项目监控堆栈在此命名空间中创建的角色分配 RBAC。**只有集群管理员才能访问这个命名空间。部署的 Helm Chart 和 Prometheus Federator 将为项目所有者（admin）、项目成员（edit）和只读成员（view）分配该命名空间的有限访问权限。** + + :::note 注意事项： + + - 项目发布命名空间会自动部署并导入到 ID 在 `.Values.helmProjectOperator.projectReleaseNamespaces.labelValue` 下指定的项目中，如果未指定，且项目注册命名空间中指定了 ProjectHelmChart，则默认为 `.Values.global.cattle.systemProjectId` 的值。 + + - 项目发布命名空​​间的孤立约定与项目注册命名空间的相同（参见上面的注释）。 + + - 如果 `.Values.projectReleaseNamespaces.enabled` 为 false，则项目发布命名空​​间与项目注册命名空间是相同的。 + + ::: + +### Helm 资源（HelmChart、HelmRelease） + +在部署 ProjectHelmChart 时，Prometheus Federator 将自动创建和管理两个子自定义资源，它们依次管理以下底层 Helm 资源： + +- HelmChart CR（通过 Operator 中的嵌入式 [k3s-io​​/helm-contoller](https://github.com/k3s-io/helm-controller) 管理）：此自定义资源会根据应用到 HelmChart CR 的变更，在触发 `helm install`、`helm upgrade` 或 `helm uninstall` 的同一命名空间中自动创建一个 Job。此 CR 会根据 ProjectHelmChart 的更改（例如，修改 `values.yaml`）或底层项目定义的更改（例如，从项目中添加或删除命名空间）自动更新。 + +:::note 重要提示： + +如果 ProjectHelmChart 没有部署或更新底层项目监控堆栈，你可以先使用此资源在 Operator / System 命名空间中创建的 Job 来检查 Helm 操作是否有问题。通常只能由**集群管理员访问**。 + +::: + +- HelmRelease CR（通过 Operator 中的嵌入式 [rancher/helm-locker](https://github.com/rancher/helm-locker) 管理）：此自定义资源会自动锁定已部署的 Helm 版本并自动覆盖对底层资源的更新，除非更改是 Helm 操作导致的（`helm install`、`helm upgrade` 或 `helm uninstall` 由 HelmChart CR 执行）。 + +:::note + +HelmRelease CR 会发出 Kubernetes 事件，用于检测底层 Helm 版本修改并将其锁定回原位。要查看这些事件，你可以使用 `kubectl describe helmrelease -n `。你还可以查看此 Operator 的日志，了解检测到更改的时间以及哪些资源被尝试更改。 + +::: + +这两种资源都是为 Operator / System 命名空间中的所有 Helm Chart 创建的，用于避免低权限用户的权限升级。 + +### 高级 Helm Project Operator 配置 + +有关高级配置的更多信息，请参阅[此页面](https://github.com/rancher/prometheus-federator/blob/main/charts/prometheus-federator/0.0.1/README.md#advanced-helm-project-operator-configuration)。 + + + +### Local 集群上的 Prometheus Federator + +Prometheus Federator 是一个资源密集型应用程序。你可以将其安装到 Local 集群（**不推荐**）。 + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md new file mode 100644 index 00000000000..ff6b6200e19 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md @@ -0,0 +1,17 @@ +--- +title: CIS 扫描指南 +--- + + + + + +- [安装 Rancher CIS Benchmark](install-rancher-cis-benchmark.md) +- [卸载 Rancher CIS Benchmark](uninstall-rancher-cis-benchmark.md) +- [运行扫描](run-a-scan.md) +- [定时运行扫描](run-a-scan-periodically-on-a-schedule.md) +- [跳过测试](skip-tests.md) +- [查看报告](view-reports.md) +- [为 Rancher CIS Benchmark 启用告警](enable-alerting-for-rancher-cis-benchmark.md) +- [为定时扫描配置告警](configure-alerts-for-periodic-scan-on-a-schedule.md) +- [为集群扫描创建自定义 Benchmark 版本](create-a-custom-benchmark-version-to-run.md) \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md new file mode 100644 index 00000000000..e6b3e08e753 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md @@ -0,0 +1,34 @@ +--- +title: 设置指南 +--- + + + + + +本文介绍如何启用 Istio 并在你的项目中使用它。 + +如果你使用 Istio 进行流量管理，则需要允许外部流量进入集群。在这种情况下，你将需要执行以下所有步骤。 + +## 先决条件 + +本指南假设你已经[安装 Rancher](../../../getting-started/installation-and-upgrade/installation-and-upgrade.md)，且已经[配置了一个单独的 Kubernetes 集群](../../new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)并要在该集群上安装 Istio。 + +集群中的节点必须满足 [CPU 和内存要求](../../../integrations-in-rancher/istio/cpu-and-memory-allocations.md)。 + +Istio 控制的工作负载和服务必须满足 [Istio 要求](https://istio.io/docs/setup/additional-setup/requirements/)。 + +## 安装 + +:::tip 快速设置提示: + +如果你不需要外部流量到达 Istio，而只想设置 Istio 以监控和跟踪集群内的流量，请跳过[设置 Istio Gateway](set-up-istio-gateway.md)和[设置 Istio 的流量管理组件](set-up-traffic-management.md)步骤。 + +::: + +1. [在集群中启用 Istio。](enable-istio-in-cluster.md) +2. [在命名空间中启用 Istio。](enable-istio-in-namespace.md) +3. [使用 Istio Sidecar 添加部署和服务。](use-istio-sidecar.md) +4. [设置 Istio Gateway。](set-up-istio-gateway.md) +5. [设置 Istio 的流量管理组件。](set-up-traffic-management.md) +6. [生成和查看流量。](generate-and-view-traffic.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/monitoring-alerting-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/monitoring-alerting-guides.md new file mode 100644 index 00000000000..16a504ae226 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/monitoring-alerting-guides.md @@ -0,0 +1,15 @@ +--- +title: Monitoring/Alerting 指南 +--- + + + + + +- [启用 Monitoring](enable-monitoring.md) +- [卸载 Monitoring](uninstall-monitoring.md) +- [为工作负载设置 Monitoring](set-up-monitoring-for-workloads.md) +- [自定义 Grafana 仪表板](customize-grafana-dashboard.md) +- [持久化 Grafana 仪表板](create-persistent-grafana-dashboard.md) +- [调试高内存用量](debug-high-memory-usage.md) + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/prometheus-federator-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/prometheus-federator-guides.md new file mode 100644 index 00000000000..7bc98a902e4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/prometheus-federator-guides.md @@ -0,0 +1,12 @@ +--- +title: Prometheus Federator 指南 +--- + + + + + +- [启用 Prometheus Federator](enable-prometheus-federator.md) +- [卸载 Prometheus Federator](uninstall-prometheus-federator.md) +- [自定义 Grafana 仪表板](customize-grafana-dashboards.md) +- [为工作负载设置 Prometheus Federator](set-up-workloads.md) \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/advanced-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/advanced-configuration.md new file mode 100644 index 00000000000..549bc6fa852 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/advanced-configuration.md @@ -0,0 +1,19 @@ +--- +title: 高级配置 +--- + + + + + +### Alertmanager + +有关配置 Alertmanager 自定义资源的信息，请参阅[此页面。](alertmanager.md) + +### Prometheus + +有关配置 Prometheus 自定义资源的信息，请参阅[此页面。](prometheus.md) + +### PrometheusRules + +有关配置 PrometheusRules 自定义资源的信息，请参阅[此页面。](prometheusrules.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md index fc12de7e488..458011a703c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md @@ -1,5 +1,5 @@ --- -title: 配置 PrometheusRule +title: PrometheusRule 配置 --- PrometheusRule 定义了一组 Prometheus 告警和/或记录规则。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md new file mode 100644 index 00000000000..3b53fe7214f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md @@ -0,0 +1,55 @@ +--- +title: 配置 +--- + + + + + +本文介绍在 Rancher UI 中配置 Monitoring V2 的一些最重要选项。 + +有关为 Prometheus 配置自定义抓取目标和规则的信息，请参阅 [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) 的上游文档。Prometheus Operator [设计文档](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md)中解释了一些最重要的自定义资源。Prometheus Operator 文档还可以帮助你设置 RBAC、Thanos 或进行自定义配置。 + +## 设置资源限制和请求 + +安装 `rancher-monitoring` 时可以配置 Monitoring 应用的资源请求和限制。有关默认限制的更多信息，请参阅[此页面](../../../reference-guides/monitoring-v2-configuration/helm-chart-options.md#配置资源限制和请求)。 + +:::tip + +在空闲集群上，Monitoring 可能会占用很多 CPU 资源。要提高性能，请关闭 Prometheus Adapter。 + +::: + +## Prometheus 配置 + +通常不需要直接编辑 Prometheus 自定义资源。 + +相反，要让 Prometheus 抓取自定义指标，你只需创建一个新的 ServiceMonitor 或 PodMonitor 来将 Prometheus 配置为抓取其他指标。 + +### ServiceMonitor 和 PodMonitor 配置 + +有关详细信息，请参阅[此页面](../../../reference-guides/monitoring-v2-configuration/servicemonitors-and-podmonitors.md)。 + +### 高级 Prometheus 配置 + +有关直接编辑 Prometheus 自定义资源（对高级用例可能有帮助）的更多信息，请参阅[此页面](advanced-configuration/prometheus.md)。 + +## Alertmanager 配置 + +Alertmanager 自定义资源通常不需要直接编辑。在常见用例中，你可以通过更新路由和接收器来管理告警。 + +路由和接收器是 Alertmanager 自定义资源配置的一部分。在 Rancher UI 中，路由（Route）和接收器（Receiver）并不是真正的自定义资源，而是 Prometheus Operator 用来将你的配置与 Alertmanager 自定义资源同步的伪自定义资源。当路由和接收器更新时，Monitoring 应用将自动更新 Alertmanager 来反映这些更改。 + +对于一些高级用例，你可能需要直接配置 Alertmanager。有关详细信息，请参阅[此页面](advanced-configuration/alertmanager.md)。 + +### 接收器 + +接收器（Receiver）用于设置通知。有关如何配置接收器的详细信息，请参阅[此页面](../../../reference-guides/monitoring-v2-configuration/receivers.md)。 + +### 路由 + +路由（Route）在通知到达接收器之前过滤它们。每条路由都需要引用一个已经配置好的接收器。有关如何配置路由的详细信息，请参阅[此页面](../../../reference-guides/monitoring-v2-configuration/routes.md)。 + +### 高级配置 + +有关直接编辑 Alertmanager 自定义资源（对高级用例可能有帮助）的更多信息，请参阅[此页面](advanced-configuration/alertmanager.md)。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cis-scans/cis-scans.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cis-scans/cis-scans.md new file mode 100644 index 00000000000..41ca4d923cb --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cis-scans/cis-scans.md @@ -0,0 +1,121 @@ +--- +title: CIS 扫描 +--- + + + + + +Rancher 可以通过运行安全扫描来检查 Kubernetes 是否按照 CIS Kubernetes Benchmark 中定义的安全最佳实践进行部署。CIS 扫描可以运行在任何 Kubernetes 集群，包括托管的 Kubernetes，例如 EKS、AKS 和 GKE。 + +`rancher-cis-benchmark` 应用使用了 kube-bench, ，这是 Aqua Security 的开源工具，用于检查集群是否符合 CIS Kubernetes Benchmark。此外，为了生成集群级别的报告，此应用使用了 Sonobuoy 来聚合报告。 + +## 关于 CIS Benchmark + +CIS（Center for Internet Security）是一个 501(c\)(3) 非营利组织，成立于 2000 年 10 月，其使命是识别、开发、验证、促进和维持网络防御的最佳实践方案，并建立和指导社区，以在网络空间中营造信任的环境。该组织总部位于纽约东格林布什，其成员包括大公司、政府机构和学术机构。 + +CIS Benchmark 是目标系统安全配置的最佳实践。CIS Benchmark 是由安全专家、技术供应商、公开和私人社区成员，以及 CIS Benchmark 开发团队共同志愿开发的。 + +在 CIS 网站上[注册](https://learn.cisecurity.org/benchmarks)以查看官方 Benchmark 文档。 + +## 关于生成的报告 + +每次扫描都会生成一份报告，你可以在 Rancher UI 中查看该报告，并以 CSV 格式下载它。 + +默认情况下使用 CIS Benchmark v1.6。 + +Benchmark 版本包含在生成的报告中。 + +Benchmark 提供两种类型的建议，分别是自动（Automated）和手动（Manual）。Benchmark 中标记为 Manual 的建议不包含在生成的报告中。 + +一些测试会被标记为“不适用”。由于 Rancher 配置 RKE 集群的方式，这些测试不会在任何 CIS 扫描中运行。有关如何审核测试结果，以及为什么某些测试会被标记为不适用，请参阅 Rancher 的 Kubernetes 对应版本的[自测指南](../../reference-guides/rancher-security/rancher-security.md#CIS-Benchmark-和自我评估)。 + +该报告包含以下信息： + +| 报告中的列 | 描述 | +|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | CIS Benchmark 的 ID 号。 | +| `description` | CIS Benchmark 测试的描述。 | +| `remediation` | 为了通过测试需要修复的内容。 | +| `state` | 测试的状态，可以是通过、失败、跳过或不适用。 | +| `node_type` | 节点角色，角色决定了在节点上运行的测试。主测试在 controlplane 节点上运行，etcd 测试在 etcd 节点上运行，节点测试在 Worker 节点上运行。 | +| `audit` | 这是 `kube-bench` 为此测试运行的审计检查。 | +| `audit_config` | 适用于审计脚本的任何配置。 | +| `test_info` | `kube-bench` 报告的测试相关信息（如果存在）。 | +| `commands` | `kube-bench` 报告的测试相关的命令（如果存在）。 | +| `config_commands` | `kube-bench` 报告的测试相关的配置数据（如果存在）。 | +| `actual_value` | 测试的实际值。如果由 `kube-bench` 报告，则会显示。 | +| `expected_result` | 测试的预期值。如果由 `kube-bench` 报告，则会显示。 | + +请参阅[集群加固指南中的表格](../../reference-guides/rancher-security/rancher-security.md)，以了解 Kubernetes、Benchmark、Rancher 以及我们的集群强化指南的版本对应关系。另外，请参阅强化指南，以获取符合 CIS 的集群的配置文件以及修复失败测试的信息。 + +## 测试配置文件 + +以下是可用的配置文件： + +- Generic CIS 1.6 +- Generic CIS 1.20 +- Generic CIS 1.23 +- RKE permissive 1.6 +- RKE hardened 1.6 +- RKE permissive 1.20 +- RKE hardened 1.20 +- RKE permissive 1.23 +- RKE hardened 1.23 +- RKE2 permissive 1.6 +- RKE2 hardened 1.6 +- RKE2 permissive 1.20 +- RKE2 hardened 1.20 +- RKE2 permissive 1.23 +- RKE2 hardened 1.23 +- K3s permissive 1.6 +- K3s hardened 1.6 +- K3s permissive 1.20 +- K3s hardened 1.20 +- K3s permissive 1.23 +- K3s hardened 1.23 +- AKS +- EKS +- GKE + +你还可以通过保存一组要跳过的测试来自定义配置文件。 + +所有配置文件都会有一组不适用的测试，CIS 扫描会跳过这些测试。RKE 集群管理 Kubernetes 的方式导致这些测试被认为不适用。 + +RKE 集群扫描配置文件有两种类型： + +- **Permissive**：此配置文件有一组要跳过的测试，跳过的原因是这些测试会在默认的 RKE Kubernetes 集群上失败。除了跳过的测试列表之外，配置文件也不会运行不适用的测试。 +- **Hardened**：此配置文件不会跳过任何测试（不适用的测试除外）。 + +EKS 和 GKE 集群扫描的配置文件基于这些集群类型特定的 CIS Benchmark 版本。 + +要通过 “Hardened” 配置文件，你需要遵从[强化指南](../../reference-guides/rancher-security/rancher-security.md#Rancher-加固指南)并使用强化指南中定义的 `cluster.yml` 来配置一个强化集群。 + +默认配置文件和支持的 CIS Benchmark 版本取决于扫描的集群类型： + +`rancher-cis-benchmark` 支持 CIS 1.6 Benchmark 版本。 + +- RKE Kubernetes 集群默认使用 RKE Permissive 1.6 配置文件。 +- EKS 和 GKE 有自己的 CIS Benchmark，由 `kube-bench` 发布。这些集群默认使用相应的测试配置文件。 +- RKE2 Kubernetes 集群默认使用 RKE2 Permissive 1.6 配置文件。 +- RKE、RKE2、EKS 和 GKE 以外的集群类型默认使用 Generic CIS 1.5 配置文件。 + +## 跳过和不适用的测试 + +有关要跳过和不适用的测试列表，请参阅[此页面](../../how-to-guides/advanced-user-guides/cis-scan-guides/skip-tests.md)。 + +目前，只有用户定义的跳过测试会在生成报告中标记为跳过。 + +如果某个默认配置文件将某个测试定义为跳过，则该测试也会标记为不适用。 + +## RBAC + +有关权限的详细信息，请参阅[此页面](rbac-for-cis-scans.md)。 + +## 配置 + +有关为扫描、配置文件和 Benchmark 版本配置自定义资源的更多信息，请参阅[此页面](configuration-reference.md)。 + +## 操作指南 + +要了解如何运行 CIS 扫描，请参阅 [CIS 扫描指南](../../how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/istio/configuration-options/configuration-options.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/istio/configuration-options/configuration-options.md new file mode 100644 index 00000000000..800b4e576da --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/istio/configuration-options/configuration-options.md @@ -0,0 +1,43 @@ +--- +title: 配置选项 +--- + + + + + +### Egress 支持 + +默认情况下，Egress 网关是禁用的，但你可以在安装或升级时使用 values.yaml 或[覆盖文件](#覆盖文件)启用它。 + +### 启用自动 Sidecar 注入 + +默认情况下，自动 sidecar 注入是禁用的。要启用此功能，请在安装或升级时在 values.yaml 中设置 `sidecarInjectorWebhook.enableNamespacesByDefault=true`。这会自动将 Istio sidecar 注入到所有已部署的新命名空间。 + +### 覆盖文件 + +覆盖文件用于为 Istio 进行更广泛的配置。它允许你更改 [IstioOperator API](https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/) 中可用的任何值。你可以自定义默认安装以满足你的需求。 + +覆盖文件将在 Istio Chart 默认安装的基础上添加配置。换言之，你不需要为安装中已定义的组件进行重新定义。 + +有关覆盖文件的更多信息，请参阅 [Istio 文档](https://istio.io/latest/docs/setup/install/istioctl/#configure-component-settings) + +### 选择器和抓取配置 + +Monitoring 应用设置了 `prometheus.prometheusSpec.ignoreNamespaceSelectors=false`，即在默认情况下跨所有命名空间进行监控。这样，你可以查看部署在具有 `istio-injection=enabled` 标签的命名空间中的资源的流量、指标和图。 + +如果你想将 Prometheus 限制为特定的命名空间，请设置 `prometheus.prometheusSpec.ignoreNamespaceSelectors=true`。完成此操作后，你需要添加其他配置来继续监控你的资源。 + +详情请参阅[本节](selectors-and-scrape-configurations.md)。 + +### 在具有 Pod 安全策略的情况下启用 Istio + +详情请参阅[本节](pod-security-policies.md)。 + +### 在 RKE2 集群上安装 Istio 的其他步骤 + +详情请参阅[本节](install-istio-on-rke2-cluster.md)。 + +### 项目网络隔离的其他步骤 + +详情请参阅[本节](project-network-isolation.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/istio/istio.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/istio/istio.md new file mode 100644 index 00000000000..42c642b4b64 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/istio/istio.md @@ -0,0 +1,140 @@ +--- +title: Istio +--- + + + + + +[Istio](https://istio.io/) 是一种开源工具，可以让 DevOps 团队更轻松地观察、控制、排查并保护复杂的微服务网络中的流量。 + +随着微服务网络的变化和增长，微服务网络之间的交互变得越来越难以管理和理解。在这种情况下，将服务网格作为单独的基础设施层是非常有用的。Istio 的服务网格可以让你在不直接更改微服务的情况下控制微服务之间的流量。 + +Rancher 与 Istio 集成，使得管理员或集群所有者可以将 Istio 交给开发者团队，然后开发者使用 Istio 执行安全策略，排查问题，或为蓝绿部署，金丝雀部署，和 A/B 测试进行流量管理。 + +此核心服务网格支持但不限于以下功能： + +- **管理流量**：例如入口和出口路由、断路、镜像。 +- **安全**：具有用于验证和授权流量和用户的资源，包括 mTLS。 +- **可观察性**：观察日志、指标和分布式流量。 + +[设置 Istio](../../how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md) 后，你可以通过 Rancher UI、`kubectl` 或 ` Istioctl` 来使用 Istio 的 controlplane 功能。 + +Istio 需要由 `cluster-admin` 设置后才能在项目中使用。 + +## Rancher 2.5 的新功能 + +Istio 已简化了整体架构。结合 Pilot、Citadel、Galley 和 sidecar injector 创建了一个单独的组件 Istiod。Node Agent 功能也已合并到 istio-agent 中。 + +以前由 Istio 安装的插件（cert-manager、Grafana、Jaeger、Kiali、Prometheus、Zipkin）现在需要单独安装。Istio 支持安装来自 Istio 项目的集成，并保持与非 Istio 项目的兼容性。 + +你仍然可以通过安装 [Rancher Monitoring](../monitoring-and-alerting/monitoring-and-alerting.md) 或安装你自己的 Prometheus operator 来使用 Prometheus 集成。Rancher 的 Istio chart 还默认安装 Kiali，确保你可以开箱即用地全面了解微服务。 + +Istio 已经脱离了使用 Helm 安装的方式，现在通过 Istioctl 二进制文件或 Istio Operator 进行安装。为了使用最简单的方式与 Istio 交互，Rancher 的 Istio 会维护一个 Helm Chart，该 Chart 使用 Istioctl 二进制文件来管理你的 Istio 安装。 + +此 Helm Chart 将在 UI 的**应用 & 市场**中提供。有权访问 Rancher Chart 应用商店的用户需要先设置 Istio，然后才能在项目中使用它。 + +## Istio 附带的工具 + +我们的 [Istio](https://istio.io/) 安装程序将 istioctl 二进制命令包装在一个 Helm chart 中，其中包括一个覆盖文件的选项，用来支持复杂的自定义配置。 + +它还包括以下内容： + +### Kiali + +Kiali 是一个全面的可视化辅助工具，用于绘制整个服务网格中的流量图。它允许你查看它们的连接方式，包括它们之间的流量速率和延迟。 + +你可以检查服务网格的运行状况，或深入查看单个组件的传入和传出请求。 + +### Jaeger + +Jaeger 是用于跟踪分布式系统的工具。我们的 Istio 安装程序包括能快速启动的一体化 [Jaeger](https://www.jaegertracing.io/) 安装。 + +请注意，这不是符合 Jaeger 生产要求的部署。此部署使用在内存中的存储组件，而 Jaeger 推荐在生产环境中使用持久存储组件。有关你所需的部署策略的更多信息，请参阅 [Jaeger 文档](https://www.jaegertracing.io/docs/latest/operator/#production-strategy)。 + +## 先决条件 + +在启用 Istio 之前，建议你先确认你的 Rancher worker 节点是否有足够的 [CPU 和内存](cpu-and-memory-allocations.md)来运行 Istio 的所有组件。 + +如果要在 RKE2 集群上安装 Istio，则需要执行一些额外的步骤。有关详细信息，请参阅[本节](#在-rke2-集群上安装-istio-的其他步骤) + +## 设置指南 + +如需了解如何设置 Istio 并在项目中使用它，请参阅[设置指南](../../how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md)。 + +## 卸载 Istio + +要从集群、命名空间或工作负载中删除 Istio 组件，请参阅[卸载 Istio](disable-istio.md)。 + +## 访问可视化 + +> 默认情况下，只有 cluster-admin 可以访问 Kiali。有关如何允许具有管理员、编辑或查看权限的角色访问它们的说明，请参阅[本节](rbac-for-istio.md)。 + +在集群中设置 Istio 后，你可以在 Rancher UI 中使用 Grafana、Prometheus 和 Kiali。 + +要访问 Grafana 和 Prometheus 可视化： + +1. 在左上角，单击 **☰ > 集群管理**。 +2. 在**集群**页面上，转到要可视化的集群，然后单击 **Explore**。 +3. 在左侧导航栏中，单击**监控**。 +4. 点击 **Grafana** 或任何其他仪表板。 + +要访问 Kiali 可视化： + +1. 在左上角，单击 **☰ > 集群管理**。 +2. 在**集群**页面上，转到要查看 Kiali 的集群，然后单击 **Explore**。 +3. 在左侧导航栏中，单击 **Istio**。 +4. 单击 **Kiali**。从这里，你可以访问**流量图**或**流量指标**选项卡，从而可视化网络指标。 + +默认情况下，prometheus 会拾取所有命名空间，并将数据用于 Kiali 图。如果你想使用不同的配置进行 prometheus 数据抓取，请参阅[选择器/抓取配置](configuration-options/selectors-and-scrape-configurations.md)。 + +你的角色决定了你对可视化的访问。只有 `cluster-admin` 角色可以使用 Grafana 和 Prometheus。默认情况下，只有 `cluster-admin` 可以使用 Kiali UI，但是 `cluster-admin` 可以通过编辑 Istio values.yaml 来允许其他角色进行访问。 + +## 架构 + +Istio 安装了一个服务网格，它使用 [Envoy](https://www.envoyproxy.io) Sidecar 代理来拦截到每个工作负载的流量。这些 sidecar 拦截并管理服务之间的通信，从而实现精细化观察并控制集群内的流量。 + +只有注入了 Istio sidecar 的工作负载可以通过 Istio 进行跟踪和控制。 + +如果命名空间启用了 Istio，部署到命名空间的新工作负载会自动具有 Istio sidecar。你需要为之前的工作负载手动启用 Istio。 + +有关 Istio sidecar 的更多信息，请参阅 [Istio sidecare-injection 文档](https://istio.io/docs/setup/kubernetes/additional-setup/sidecar-injection/)。有关 Istio 架构的更多信息，请参阅 [Istio 架构文档](https://istio.io/latest/docs/ops/deployment/architecture/)。 + +### 多个 Ingress + +默认情况下，每个 Rancher 配置的集群都有一个 NGINX Ingress Controller 来允许流量进入集群。Istio 还在 `istio-system` 命名空间中默认安装一个 Ingress Gateway。因此，你的集群将有两个 ingress。 + +![启用 Istio 的集群可以有两个 ingress，分别是默认的 Nginx ingress 和默认的 Istio controller](/img/istio-ingress.svg) + +可以通过[覆盖文件](configuration-options/configuration-options.md#覆盖文件)来启用其他 Istio Ingress Gateway。 + +### Egress 支持 + +默认情况下，Egress 网关是禁用的，但你可以在安装或升级时使用 values.yaml 或[覆盖文件](configuration-options/configuration-options.md#覆盖文件)启用它。 + +## 在 RKE2 集群上安装 Istio 的其他步骤 + +要在 RKE2 集群上安装 Istio，请按照[步骤](configuration-options/install-istio-on-rke2-cluster.md)进行操作。 + +## 在离线环境中升级 Istio + +现在，Istio Pod 安全策略默认启用。新值 `installer.releaseMirror.enabled` 已添加到 rancher-istio Chart 中，以启用和禁用支持离线升级的 Server。请注意，`installer.releaseMirror.enabled` 默认设置为 `false`。你可以在安装或升级时根据需要设置该值。按照以下步骤执行： + +1. 在 Rancher UI 中配置离线 Rancher 实例和离线自定义集群。 +2. 在集群中安装 Monitoring：**Cluster Explorer > Apps & Marketplace > Charts > Monitoring**。 +3. 将 Istio 所需的所有镜像拉入在离线环境中使用的私有镜像仓库。 +4. 在集群中安装 Istio：**Cluster Explorer > Apps & Marketplace > Charts > Istio**。 + +:::note + +你可以在新安装的 Istio 上启用 [Jaeger](https://www.jaegertracing.io/) 和 [Kiali](https://kiali.io/)。为确保 Jaeger 和 Kiali 正常工作，请在安装期间将 `values.yaml` 中的 `installer.releaseMirror.enabled` 设置为 `true`。 + +::: + +5. 升级 Istio。 + +:::caution + +如果你还没有执行操作，请设置 `installer.releaseMirror.enabled=true` 以升级 Istio。 + +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/logging/custom-resource-configuration/custom-resource-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/logging/custom-resource-configuration/custom-resource-configuration.md new file mode 100644 index 00000000000..d719b72fe1d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/logging/custom-resource-configuration/custom-resource-configuration.md @@ -0,0 +1,12 @@ +--- +title: 自定义资源配置 +--- + + + + + +通过以下自定义资源配置 logging： + +- [Flow 和 ClusterFlow](flows-and-clusterflows.md) +- [Output 和 ClusterOutput](outputs-and-clusteroutputs.md) \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/logging/logging.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/logging/logging.md new file mode 100644 index 00000000000..dcc65014fbd --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/logging/logging.md @@ -0,0 +1,115 @@ +--- +title: Rancher Logging 集成 +description: Rancher 集成了主流的日志服务。了解集成日志服务的要求和优势，并在你的集群上启用 Logging。 +--- + + + + + +现在，Rancher 的日志管理由 [Logging operator](https://kube-logging.github.io/docs/) 提供支持，它取代了以前的内部解决方案。 + +## 启用 Logging + +你可以转到**应用**页面并安装 Logging 应用程序，从而为 Rancher 管理的集群启用 Logging： + +1. 转到要安装 Logging 的集群，然后单击 **Apps**。 +1. 点击 **Logging** 应用。 +1. 滚动到 Helm Chart README 的底部，然后单击**安装**。 + +**结果**：Logging 应用已部署到 `cattle-logging-system` 命名空间中。 + +## 卸载 Logging + +1. 转到要安装 Logging 的集群，然后单击 **Apps**。 +1. 点击**已安装的应用**。 +1. 转到 `cattle-logging-system` 命名空间并选中 `rancher-logging` 和 `rancher-logging-crd` 框。 +1. 单击**删除**。 +1. 确认**删除**。 + +**结果**：已卸载 `rancher-logging`。 + +## 架构 + +有关 Logging 应用程序工作原理的更多信息，请参阅[本节](logging-architecture.md)。 + +## RBAC + +Rancher Logging 有两个角色，分别是 `logging-admin` 和 `logging-view`。有关如何以及何时使用这些角色的更多信息，请参阅[此页面](rbac-for-logging.md)。 + +## 配置 Logging 自定义资源 + +要管理 `Flows`、`ClusterFlows`、`Outputs` 和 `ClusterOutputs`： + +1. 在左上角，单击 **☰ > 集群管理**。 +2. 在**集群**页面上，转到要配置 Logging 自定义资源的集群，然后单击 **Explore**。 +3. 在左侧导航栏中，单击 **Logging**。 + +### Flows 和 ClusterFlows + +有关配置 `Flows` 和 `ClusterFlows` 的帮助，请参阅[此页面](custom-resource-configuration/flows-and-clusterflows.md)。 + +### Outputs 和 ClusterOutputs + +有关配置 `Outputs` 和 `ClusterOutputs` 的帮助，请参阅[此页面](custom-resource-configuration/outputs-and-clusteroutputs.md)。 + +## 配置 Logging Helm Chart + +有关在安装或升级 Logging 应用程序时可配置的选项，请参阅[此页面](logging-helm-chart-options.md)。 + +### Windows 支持 + +你可以从 Windows 节点[启用 Logging](logging-helm-chart-options.md#启用禁用-windows-节点-logging)。 + +### 使用自定义 Docker 根目录 + +有关使用自定义 Docker 根目录的详细信息，请参阅[本节](logging-helm-chart-options.md#使用自定义-docker-根目录)。 + +### 处理污点和容忍度 + +有关如何在 Logging 应用程序中使用污点和容忍度的信息，请参阅[此页面](taints-and-tolerations.md)。 + +### 在 SELinux 上使用 Logging V2 + +有关在启用了 SELinux 的节点上使用 Logging 应用程序的信息，请参阅[本节](logging-helm-chart-options.md#启用-logging-应用程序以使用-selinux)。 + +### 其他日志来源 + +默认情况下，Rancher 会收集所有类型集群的 controlplane 组件和节点组件的日志。在某些情况下，也会收集其他日志。有关详细信息，请参阅[本节](logging-helm-chart-options.md#其他日志来源)。 + +## 故障排除 + +### 日志缓冲区导致 Pod 过载 + +根据你的配置，默认缓冲区大小可能太大并导致 Pod 故障。减少负载的一种方法是降低记录器的刷新间隔。这可以防止日志溢出缓冲区。你还可以添加更多刷新线程来处理大量日志试图同时填充缓冲区的情况。 + +有关如何配置日志缓冲区来满足企业需求的更完整说明，请参阅[缓冲区](https://kube-logging.github.io/docs/configuration/plugins/outputs/buffer/)和 [Fluentd 配置](https://kube-logging.github.io/docs/logging-infrastructure/fluentd/)的官方 Logging Operator 文档。 + +### `cattle-logging` 命名空间正在重新创建 + +如果你的集群之前在旧版 Rancher UI 的全局视图中部署了 Logging，`cattle-logging` 命名空间可能会不断被重新创建。 + +要解决这个问题，你可以将所有 `clusterloggings.management.cattle.io` 和 `projectloggings.management.cattle.io` 自定义资源从管理集群中针对该集群的命名空间中删除。 +这些自定义资源会导致 Rancher 在下游集群中创建 `cattle-logging` 命名空间（如果不存在）。 + +集群命名空间与集群 ID 匹配，因此我们需要找到每个集群的集群 ID。 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面上，转到要获取 ID 的集群，然后单击 **Explore**。 +2. 从以下其中一个 URL 中复制 `` 的内容。`` 是集群命名空间名称。 + +```bash +# Cluster Management UI +https:///c// + +# Cluster Dashboard +https:///dashboard/c// +``` + +现在我们有了 `` 命名空间，我们可以删除导致 `cattle-logging` 不断重新创建的自定义资源。 +*警告*：请确保当前未使用 Logging （从旧版 Rancher UI 全局视图中安装的版本）。 + +```bash +kubectl delete crd clusterloggings.management.cattle.io -n +kubectl delete crd projectloggings.management.cattle.io -n +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md new file mode 100644 index 00000000000..cebb2e4323f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md @@ -0,0 +1,106 @@ +--- +title: 监控和告警 +description: Prometheus 允许你查看来自不同 Rancher 和 Kubernetes 对象的指标。了解监控范围以及如何启用集群监控 +--- + + + + + +你可以使用 `rancher-monitoring` 应用，将业界领先的开源监控和告警解决方案快速部署到你的集群中。 + +在 Rancher v2.5 中引入的 `rancher-monitoring` operator 由 [Prometheus](https://prometheus.io/)、[Grafana](https://grafana.com/grafana/)、[Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)， [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) 和 [Prometheus adapter](https://github.com/DirectXMan12/k8s-prometheus-adapter) 提供支持。 + +有关在 Rancher v2.2 到 v2.4 中可用的 V1 监控和告警的信息，请参阅 Rancher v2.0 到 v2.4 文档中的[集群监控](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/cluster-monitoring/cluster-monitoring.md)，[告警](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/cluster-alerts/cluster-alerts.md)，[通知](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/notifiers.md)和[工具](/versioned_docs/version-2.0-2.4/reference-guides/rancher-project-tools/rancher-project-tools.md)。 + +使用 `rancher-monitoring` 应用程序，你可以快速部署领先的开源监控和告警解决方案到你的集群上。 + +### 功能 + +Prometheus 支持查看 Rancher 和 Kubernetes 对象的指标。通过使用时间戳，Prometheus 能让你通过 Rancher UI 或 Grafana（与 Prometheus 一起部署的分析查看平台）以更容易阅读的图表和视觉形式来查询和查看这些指标。 + +通过查看 Prometheus 从集群的 controlplane、节点和 deployment 中抓取的数据，你可以随时了解集群中发生的所有事件。然后，你可以使用这些分析来更好地运行你的环境，例如在系统紧急情况发生之前阻止它们、制定维护策略，或恢复崩溃的服务器。 + +Monitoring 应用： + +- 监控集群节点、Kubernetes 组件和软件部署的状态和进程。 +- 根据 Prometheus 收集的指标定义告警。 +- 创建自定义 Grafana 仪表板。 +- 使用 Prometheus Alertmanager 通过电子邮件、Slack、PagerDuty 等配置告警通知。 +- 根据 Prometheus 收集的指标，将预先计算的、经常需要的，或计算成本高的表达式定义为新的时间序列。 +- 通过 Prometheus Adapter，将从 Prometheus 收集的指标公开给 Kubernetes Custom Metrics API，以便在 HPA 中使用。 + +有关监控组件如何协同工作的说明，请参阅 [Monitoring 工作原理](how-monitoring-works.md)。 + +## 默认组件和部署 + +### 内置仪表板 + +默认情况下，监控应用将 Grafana 仪表板（由 [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) 项目策划）部署到集群上。 + +它还部署一个 Alertmanager UI 和一个 Prometheus UI。有关这些工具的更多信息，请参见[内置仪表板](built-in-dashboards.md)。 + +### 默认指标 Exporter + +默认情况下，Rancher Monitoring 会部署 Exporter（例如 [node-exporter](https://github.com/prometheus/node_exporter) 和 [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)）。 + +这些默认 Exporter 会自动从 Kubernetes 集群的所有组件（包括工作负载）中抓取 CPU 和内存的指标。 + +### 默认告警 + +Monitoring 应用会默认部署一些告警。要查看默认告警，请转到 [Alertmanager UI](built-in-dashboards.md#alertmanager-ui) 并单击**展开所有组**。 + +### Rancher UI 中公开的组件 + +有关 Rancher UI 中公开的监控组件列表，以及编辑它们的常见用例，请参阅[本节](how-monitoring-works.md#rancher-ui-中公开的组件)。 + +## RBAC + +有关配置 monitoring 访问权限的信息，请参阅[此页面](rbac-for-monitoring.md)。 + +:::note + +Rancher 和 Project 的读取权限并不一定适用于监控相关资源. 查看 [monitoring-ui-view](rbac-for-monitoring.md#其他监控角色) 获取更多详细信息. + +::: + +## 指南 + +- [启用 monitoring](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md) +- [卸载 monitoring](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/uninstall-monitoring.md) +- [Monitoring 工作负载](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/set-up-monitoring-for-workloads.md) +- [自定义 Grafana 仪表板](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/customize-grafana-dashboard.md) +- [持久化 Grafana 仪表板](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/create-persistent-grafana-dashboard.md) +- [调试高内存使用率](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/debug-high-memory-usage.md) + +## 配置 + +### 在 Rancher 中配置 Monitoring 资源 + +此处的配置参考假设你已经熟悉 monitoring 组件的协同工作方式。如需更多信息，请参阅 [monitoring 的工作原理](how-monitoring-works.md)。 + +- [ServiceMonitor 和 PodMonitor](../../reference-guides/monitoring-v2-configuration/servicemonitors-and-podmonitors.md) +- [接收器](../../reference-guides/monitoring-v2-configuration/receivers.md) +- [路由](../../reference-guides/monitoring-v2-configuration/routes.md) +- [PrometheusRule](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md) +- [Prometheus](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheus.md) +- [Alertmanager](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/alertmanager.md) + +### 配置 Helm Chart 选项 + +有关 `rancher-monitoring` Chart 选项的更多信息，包括设置资源限制和请求的选项，请参阅 [Helm Chart 选项](../../reference-guides/monitoring-v2-configuration/helm-chart-options.md)。 + +## Windows 集群支持 + +如果 Monitoring 部署到 RKE1 Windows 集群，Monitoring V2 将自动部署 [windows-exporter](https://github.com/prometheus-community/windows_exporter) DaemonSet 并设置 ServiceMonitor，以从每个部署的 Pod 中收集指标。这将使用 `windows_` 指标填充 Prometheus，这些指标与 [node_exporter](https://github.com/prometheus/node_exporter) 为 Linux 主机导出的 `node_` 指标类似。 + +为了能够为 Windows 完全部署 Monitoring V2，你的所有 Windows 主机都必须至少具有 v0.1.0 的 [wins](https://github.com/rancher/wins) 版本。 + +有关如何在现有 Windows 主机上升级 wins 版本的更多信息，请参阅 [Windows 集群对 Monitoring V2 的支持](windows-support.md)。 + + +## 已知问题 + +有一个[已知问题](https://github.com/rancher/rancher/issues/28787#issuecomment-693611821)，即 K3s 集群需要的内存超过分配的默认内存。如果你在 K3s 集群上启用 Monitoring，将 `prometheus.prometheusSpec.resources.memory.limit` 设置为 2500 Mi，并将 `prometheus.prometheusSpec.resources.memory.request` 设置为 1750 Mi。 + +如需获取意见和建议，请参阅[调试高内存使用情况](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/debug-high-memory-usage.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md new file mode 100644 index 00000000000..1110b88e03a --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md @@ -0,0 +1,15 @@ +--- +title: Monitoring V2 配置 +--- + + + + + +以下将说明在 Rancher 中如何配置 Monitoring V2 的基本重要选项： + +- [接收器配置](receivers.md) +- [路由配置](routes.md) +- [ServiceMonitor 和 PodMonitor 配置](servicemonitors-and-podmonitors.md) +- [Helm Chart 选项](helm-chart-options.md) +- [示例](examples.md) \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/prometheus-federator/prometheus-federator.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/prometheus-federator/prometheus-federator.md new file mode 100644 index 00000000000..6e4c4bd8ac3 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/prometheus-federator/prometheus-federator.md @@ -0,0 +1,109 @@ +--- +title: Prometheus Federator +--- + + + + + +Prometheus Federator（也称为 Project Monitoring V2）基于 [rancher/helm-project-operator](https://github.com/rancher/helm-project-operator) 部署一个 Helm Project Operator。该 Operator 管理 Helm Chart 的部署，每个 Operator 都包含一个 Project Monitoring Stack，而每个堆栈都包含： + +- [Prometheus](https://prometheus.io/)（由 [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) 在外部管理） +- [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)（由 [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) 在外部管理） +- [Grafana](https://github.com/helm/charts/tree/master/stable/grafana)（通过嵌入式 Helm Chart 部署） +- 基于 [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus/) 社区策划资源集合的默认 PrometheusRules 和 Grafana 仪表板 +- 监视已部署资源的默认 ServiceMonitor + +:::note 重要提示： + +Prometheus Federator 适合在已安装 Prometheus Operator CRD 的集群中与现有的 Prometheus Operator Deployment 一起部署。 + +::: + +## Operator 工作原理 + +1. 在部署此 Chart 时，用户可以创建 ProjectHelmCharts CR，并在**项目 Registration 命名空间 (`cattle-project-`)** 中将 `spec.helmApiVersion` 设置为 `monitoring.cattle.io/v1alpha1`（在 Rancher UI 中也称为“项目监控”）。 +2. 在看到每个 ProjectHelmChartCR 时，Operator 会代表项目所有者在**项目 Release 命名空间 (`cattle-project--monitoring`)** 中自动部署一个 Project Prometheus 堆栈（基于 ProjectHelmChart 控制器在 **Operator / System Namespace** 中创建的 HelmChart CR 和 HelmRelease CR）。 +3. RBAC 将自动分配到项目 Release 命名空间中，从而允许用户查看 Prometheus、Alertmanager 以及已部署的 Project Monitoring Stack 的 Grafana UI（基于在项目 Registration 命名空间上针对[面向用户的默认 Kubernetes 角色](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles)定义的 RBAC）。有关详细信息，请参阅[配置 RBAC](rbac.md)。 + +### 什么是项目？ + +在 Prometheus Federator 中，项目是一组可以由 `metav1.LabelSelector` 标识的命名空间。默认情况下，用于标识项目的标签是 `field.cattle.io/projectId`，该标签用于标识给定 Rancher 项目中包含的命名空间。 + +### 配置由 ProjectHelmChart 创建的 Helm 版本 + +此 ProjectHelmChart 资源的 `spec.values` 对应于为底层 Helm Chart 配置的 `values.yaml` 覆盖，该 Helm Chart 是 Operator 代表用户部署的。要查看底层 Chart 的 `values.yaml` 规范，你可以选择以下其中一种方式： + +- 查看位于 [`charts/rancher-project-monitoring` 中的 `rancher/prometheus-federator`](https://github.com/rancher/prometheus-federator/blob/main/charts/rancher-project-monitoring) 的 Chart 定义（Chart 版本会绑定到 Operator 版本）。 +- 查找在每个项目 Registration 命名空间中自动创建的名为 `monitoring.cattle.io.v1alpha1` 的 ConfigMap，其中包含用于配置 Chart（直接嵌入到 `prometheus-federator` 二进制文件中）的`values.yaml` 和 `questions.yaml`。 + +### 命名空间 + +Prometheus Federator 是基于 [rancher/helm-project-operator](https://github.com/rancher/helm-project-operator) 的 Project Operator，Prometheus Federator 提供了三类命名空间供 Operator 查找： + +1. **Operator / System 命名空间**：部署 Operator 的命名空间（例如 `cattle-monitoring-system`）。此命名空间将包含该 Operator 监视的所有 ProjectHelmChart 的所有 HelmChart 和 HelmRelease。**只有集群管理员才能访问此命名空间**。 + +2. **项目 Registration 命名空间 (`cattle-project-`)**：Operator 在这些命名空间中监视 ProjectHelmChart。对于在项目发布命名空​​间中创建的自动分配的 RBAC，应用于此命名空间的 RoleBinding 和 ClusterRoleBinding 也会作为 RBAC 的真实来源。有关详细信息，请参阅 [RBAC 页面](rbac.md)。**项目所有者（admin）、项目成员（edit）和只读成员（view）应该有权访问此命名空间。** + + :::note 注意事项： + + - 如果提供了 `.Values.global.cattle.projectLabel`（默认设置为 `field.cattle.io/projectId`），则 Operator 会自动生成项目注册命名空间，并将它导入到命名空间绑定的项目中。换言之，如果观察到至少一个带有该标签的命名空间，则 Operator 会创建一个项目 Registration 命名空间。除非出现以下两种情况，否则 Operator 不会让这些命名空间被删除。第一种情况是带有该标签的所有命名空间都消失了（例如，这是该项目中的最后一个命名空间，在这种情况下，命名空间将标有标签 `"helm.cattle.io/helm-project-operator-orphaned": "true"`，表示可以删除）。第二种情况是由于项目 ID 是在 `.Values.helmProjectOperator.otherSystemProjectLabelValues` 下提供的（用作项目的拒绝名单），导致 Operator 不再监视该项目。这些命名空间不会被自动删除，这样能避免破坏用户数据。如果需要，建议用户在创建或删除项目时手动清理这些命名空间。 + + - 如果未提供 `.Values.global.cattle.projectLabel`，则 Operator / System 命名空间也是项目注册命名空间。 + + ::: + +3. **项目发布命名空​​间（`cattle-project--monitoring`）**：Operator 代表 ProjectHelmChart 在其中部署项目监控堆栈的命名空间集。Operator 还将根据在项目 Registration 命名空间中找到的绑定，自动为项目监控堆栈在此命名空间中创建的角色分配 RBAC。**只有集群管理员才能访问这个命名空间。部署的 Helm Chart 和 Prometheus Federator 将为项目所有者（admin）、项目成员（edit）和只读成员（view）分配该命名空间的有限访问权限。** + + :::note 注意事项： + + - 项目发布命名空间会自动部署并导入到 ID 在 `.Values.helmProjectOperator.projectReleaseNamespaces.labelValue` 下指定的项目中，如果未指定，且项目注册命名空间中指定了 ProjectHelmChart，则默认为 `.Values.global.cattle.systemProjectId` 的值。 + + - 项目发布命名空​​间的孤立约定与项目注册命名空间的相同（参见上面的注释）。 + + - 如果 `.Values.projectReleaseNamespaces.enabled` 为 false，则项目发布命名空​​间与项目注册命名空间是相同的。 + + ::: + +### Helm 资源（HelmChart、HelmRelease） + +在部署 ProjectHelmChart 时，Prometheus Federator 将自动创建和管理两个子自定义资源，它们依次管理以下底层 Helm 资源： + +- HelmChart CR（通过 Operator 中的嵌入式 [k3s-io​​/helm-contoller](https://github.com/k3s-io/helm-controller) 管理）：此自定义资源会根据应用到 HelmChart CR 的变更，在触发 `helm install`、`helm upgrade` 或 `helm uninstall` 的同一命名空间中自动创建一个 Job。此 CR 会根据 ProjectHelmChart 的更改（例如，修改 `values.yaml`）或底层项目定义的更改（例如，从项目中添加或删除命名空间）自动更新。 + +:::note 重要提示： + +如果 ProjectHelmChart 没有部署或更新底层项目监控堆栈，你可以先使用此资源在 Operator / System 命名空间中创建的 Job 来检查 Helm 操作是否有问题。通常只能由**集群管理员访问**。 + +::: + +- HelmRelease CR（通过 Operator 中的嵌入式 [rancher/helm-locker](https://github.com/rancher/helm-locker) 管理）：此自定义资源会自动锁定已部署的 Helm 版本并自动覆盖对底层资源的更新，除非更改是 Helm 操作导致的（`helm install`、`helm upgrade` 或 `helm uninstall` 由 HelmChart CR 执行）。 + +:::note + +HelmRelease CR 会发出 Kubernetes 事件，用于检测底层 Helm 版本修改并将其锁定回原位。要查看这些事件，你可以使用 `kubectl describe helmrelease -n `。你还可以查看此 Operator 的日志，了解检测到更改的时间以及哪些资源被尝试更改。 + +::: + +这两种资源都是为 Operator / System 命名空间中的所有 Helm Chart 创建的，用于避免低权限用户的权限升级。 + +### 高级 Helm Project Operator 配置 + +有关高级配置的更多信息，请参阅[此页面](https://github.com/rancher/prometheus-federator/blob/main/charts/prometheus-federator/0.0.1/README.md#advanced-helm-project-operator-configuration)。 + + + +### Local 集群上的 Prometheus Federator + +Prometheus Federator 是一个资源密集型应用程序。你可以将其安装到 Local 集群（**不推荐**）。 + From 1ac5230a1c9f6b699cd037a8bcf19123ca76169c Mon Sep 17 00:00:00 2001 From: LiuYan <361112237@qq.com> Date: Thu, 11 Apr 2024 22:30:36 +0800 Subject: [PATCH 45/67] Update missing Chinese translation (#1207) * Latest version add missing Chinese translation files * Version 2.8 add missing Chinese translation files --- .../current.json | 24 +- .../about-rke1-templates.md | 130 ++++++++ .../infrastructure-setup.md | 11 + .../kubernetes-cluster-setup.md | 11 + ...checklist-for-production-ready-clusters.md | 52 ++++ .../kubernetes-clusters-in-rancher-setup.md | 78 +++++ .../migrate-to-out-of-tree-amazon.md | 194 ++++++++++++ .../migrate-to-out-of-tree-vsphere.md | 111 +++++++ .../set-up-cloud-providers.md | 47 +++ .../use-windows-clusters.md | 283 +++++++++++++++++ .../access-clusters/access-clusters.md | 65 ++++ .../create-kubernetes-persistent-storage.md | 79 +++++ .../install-cluster-autoscaler.md | 28 ++ .../manage-clusters/manage-clusters.md | 36 +++ .../provisioning-storage-examples.md | 15 + .../_cluster-capabilities-table.md | 10 +- .../version-2.8.json | 23 +- .../about-rke1-templates.md | 130 ++++++++ .../infrastructure-setup.md | 11 + .../kubernetes-cluster-setup.md | 11 + ...checklist-for-production-ready-clusters.md | 52 ++++ .../kubernetes-clusters-in-rancher-setup.md | 78 +++++ .../migrate-to-out-of-tree-amazon.md | 194 ++++++++++++ .../migrate-to-out-of-tree-vsphere.md | 111 +++++++ .../set-up-cloud-providers.md | 47 +++ .../use-windows-clusters.md | 284 ++++++++++++++++++ .../access-clusters/access-clusters.md | 65 ++++ .../create-kubernetes-persistent-storage.md | 79 +++++ .../install-cluster-autoscaler.md | 28 ++ .../manage-clusters/manage-clusters.md | 34 +++ .../provisioning-storage-examples.md | 15 + .../_cluster-capabilities-table.md | 10 +- 32 files changed, 2321 insertions(+), 25 deletions(-) create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-amazon.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/install-cluster-autoscaler/install-cluster-autoscaler.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/manage-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/provisioning-storage-examples.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-amazon.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/install-cluster-autoscaler/install-cluster-autoscaler.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/manage-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/provisioning-storage-examples.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current.json b/i18n/zh/docusaurus-plugin-content-docs/current.json index 4ce0df781ac..5a6ef279517 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current.json +++ b/i18n/zh/docusaurus-plugin-content-docs/current.json @@ -124,32 +124,40 @@ "description": "The label for category Infrastructure Setup in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Kubernetes Clusters in Rancher Setup": { - "message": "Rancher 设置中的 Kubernetes 集群", + "message": "Rancher 中的 Kubernetes 集群设置", "description": "The label for category Kubernetes Clusters in Rancher Setup in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Checklist for Production-Ready Clusters": { "message": "生产就绪集群检查清单", "description": "The label for category Checklist for Production-Ready Clusters in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Set Up Clusters from Hosted Kubernetes Providers": { + "sidebar.tutorialSidebar.category.Setting up Clusters from Hosted Kubernetes Providers": { "message": "通过托管 Kubernetes 提供商设置集群", "description": "The label for category Set Up Clusters from Hosted Kubernetes Providers in sidebar tutorialSidebar" }, + "sidebar.tutorialSidebar.category.Launching Kubernetes on Windows Clusters": { + "message": "在 Windows 集群上启动 Kubernetes", + "description": "The label for category Launching Kubernetes on Windows Clusters in sidebar tutorialSidebar" + }, "sidebar.tutorialSidebar.category.Use Windows Clusters": { "message": "使用 Windows 集群", "description": "The label for category Use Windows Clusters in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Set Up Cloud Providers": { + "sidebar.tutorialSidebar.category.Setting up Cloud Providers": { "message": "设置 Cloud Provider", - "description": "The label for category Set Up Cloud Providers in sidebar tutorialSidebar" + "description": "The label for category Setting up Cloud Providers in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Launch Kubernetes with Rancher": { + "sidebar.tutorialSidebar.category.Migrate to an Out-of-tree Cloud Provider": { + "message": "迁移到树外云提供商", + "description": "The label for category Migrate to an Out-of-tree Cloud Provider in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Launching Kubernetes with Rancher": { "message": "使用 Rancher 启动 Kubernetes", "description": "The label for category Launch Kubernetes with Rancher in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Use New Nodes in an Infra Provider": { - "message": "在基础设施提供商中使用新节点", - "description": "The label for category Use New Nodes in an Infra Provider in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Launching New Nodes in an Infra Provider": { + "message": "在基础设施提供商中启动新节点", + "description": "The label for category Launching New Nodes in an Infra Provider in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.vSphere": { "message": "vSphere", diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md new file mode 100644 index 00000000000..de9bc25d31a --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md @@ -0,0 +1,130 @@ +--- +title: RKE 模板 +--- + + + + + +RKE 模板旨在让 DevOps 和安全团队标准化和简化 Kubernetes 集群创建的流程。 + +RKE 的全称是 [Rancher Kubernetes Engine](https://rancher.com/docs/rke/latest/en/)，它是 Rancher 用来配置 Kubernetes 集群的工具。 + +随着 Kubernetes 越来越受欢迎，管理更多小型集群逐渐成为趋势。如果你想要创建大量集群，对集群进行一致管理尤为重要。多集群管理面临着安全和附件配置执行的挑战，在将集群移交给最终用户之前，这些配置需要标准化。 + +RKE 模板有助于标准化这些配置。无论是使用 Rancher UI、Rancher API 还是自动化流程创建的集群，Rancher 都将保证从 RKE 集群模板创建的每个集群在生成方式上是一致的。 + +管理员可以控制最终用户能更改的集群选项。RKE 模板还可以与特定的用户和组共享，以便管理员可以为不同的用户集创建不同的 RKE 模板。 + +如果集群是使用 RKE 模板创建的，则不能让集群使用另一个 RKE 模板。你只能将集群更新为同一模板的新版本。 + +你可以[将现有集群的配置保存为 RKE 模板](apply-templates.md#将现有集群转换为使用-rke-模板)。这样，只有模板更新后才能更改集群的设置。新模板还可用于启动新集群。 + +RKE 模板的核心功能允许 DevOps 和安全团队： + +- 标准化集群配置并确保按照最佳实践创建 Rancher 配置的集群 +- 配置集群时，防止用户做出不明智的选择 +- 与不同的用户和组共享不同的模板 +- 将模板的所有权委托给受信任的用户进行更改 +- 控制哪些用户可以创建模板 +- 要求用户使用模板来创建集群 + +## 可配置的设置 + +RKE 模板可以在 Rancher UI 中创建或以 YAML 格式定义。当你使用 Rancher 从基础设施提供商配置自定义节点或一般节点时，它们可以指定为相同的参数： + +- 云提供商选项 +- Pod 安全选项 +- 网络提供商 +- Ingress Controller +- 网络安全配置 +- 网络插件 +- 私有镜像仓库 URL 和凭证 +- 附加组件 +- Kubernetes 选项，包括 kube-api、kube-controller、kubelet 和服务等 Kubernetes 组件的配置 + +RKE 模板的[附加组件](#附加组件)的功能特别强大，因为它允许多种自定义选项。 + +## RKE 模板的范围 + +Rancher 配置的集群支持 RKE 模板。模板可用于配置自定义集群或由基础设施提供商启动的集群。 + +RKE 模板用于定义 Kubernetes 和 Rancher 设置。节点模板负责配置节点。有关如何将 RKE 模板与硬件结合使用的参考，请参阅 [RKE 模板和硬件](infrastructure.md)。 + +可以从头开始创建 RKE 模板来预先定义集群配置。它们可以用于启动新集群，也可以从现有的 RKE 集群导出模板。 + +现有集群的设置可以[保存为 RKE 模板](apply-templates.md#将现有集群转换为使用-rke-模板)。这会创建一个新模板并将集群设置绑定到该模板。这样，集群只有在[模板更新](manage-rke1-templates.md#更新模板)的情况下才能[使用新版本的模板](manage-rke1-templates.md#升级集群以使用新的模板修订版)进行升级。新模板也可以用来创建新集群。 + + +## 示例场景 +如果一个组织同时拥有普通和高级 Rancher 用户，管理员可能希望为高级用户提供更多用于集群创建的选项，并限制普通用户的选项。 + +这些[示例场景](example-use-cases.md)描述组织如何使用模板来标准化集群创建。 + +示例场景包括： + +- **强制执行模板**：如果希望所有 Rancher 配置的新集群都具有某些设置，管理员可能想要[为每个用户强制执行一项或多项模板设置](example-use-cases.md#强制执行模板设置)。 +- **与不同的用户共享不同的模板**：管理员可以为[普通用户和高级用户提供不同的模板](example-use-cases.md#普通用户和高级用户模板)。这样，普通用户会有更多限制选项，而高级用户在创建集群时可以使用更多选项。 +- **更新模板设置**：如果组织的安全和 DevOps 团队决定将最佳实践嵌入到新集群所需的设置中，这些最佳实践可能会随着时间而改变。如果最佳实践发生变化，[可以将模板更新为新版本](example-use-cases.md#更新模板和集群)，这样，使用模板创建的集群可以[升级到模板的新版本](manage-rke1-templates.md#升级集群以使用新的模板修订版)。 +- **共享模板的所有权**：当模板所有者不再想要维护模板或想要共享模板的所有权时，此方案描述了如何[共享模板所有权](example-use-cases.md#允许其他用户控制和共享模板)。 + +## 模板管理 + +创建 RKE 模板时，可以在 Rancher UI 中的**集群管理**下的 **RKE 模板**中使用模板。创建模板后，你将成为模板所有者，这将授予你修改和共享模板的权限。你可以与特定用户或组共享 RKE 模板，也可以公开模板。 + +管理员可以开启模板强制执行，要求用户在创建集群时始终使用 RKE 模板。这使管理员可以保证 Rancher 总是创建指定配置的集群。 + +RKE 模板更新通过修订系统处理。如果要更改或更新模板，请创建模板的新版本。然后，可以将使用旧版本模板创建的集群升级到新模板修订版。 + +在 RKE 模板中，模板所有者可以限制设置的内容，也可以打开设置以供最终用户选择值。它们的差别体现在，创建模板时，Rancher UI 中的每个设置上的**允许用户覆盖**标示。 + +对于无法覆盖的设置，最终用户将无法直接编辑它们。为了让用户使用这些设置的不同选项，RKE 模板所有者需要创建 RKE 模板的新版本，这将允许用户升级和更改该选项。 + +本节中的文件解释了 RKE 模板管理的细节： + +- [获取创建模板的权限](creator-permissions.md) +- [创建和修改模板](manage-rke1-templates.md) +- [强制执行模板设置](enforce-templates.md#强制新集群使用-rke-模板) +- [覆盖模板设置](override-template-settings.md) +- [与集群创建者共享模板](access-or-share-templates.md#与特定用户或组共享模板) +- [共享模板的所有权](access-or-share-templates.md#共享模板所有权) + +你可以参见此[模板的示例 YAML 文件](../../../../reference-guides/rke1-template-example-yaml.md)作为参考。 + +## 应用模板 + +你可以使用你自己创建的模板来[创建集群](apply-templates.md#使用-rke-模板创建集群)，也可以使用[与你共享的模板](access-or-share-templates.md)来创建集群。 + +如果 RKE 模板所有者创建了模板的新版本，你可以[将你的集群升级到该版本](apply-templates.md#更新使用-rke-模板创建的集群)。 + +可以从头开始创建 RKE 模板来预先定义集群配置。它们可以用于启动新集群，也可以从现有的 RKE 集群导出模板。 + +你可以[将现有集群的配置保存为 RKE 模板](apply-templates.md#将现有集群转换为使用-rke-模板)。这样，只有模板更新后才能更改集群的设置。 + +## 标准化硬件 + +RKE 模板的目的是标准化 Kubernetes 和 Rancher 设置。如果你还想标准化你的基础设施，一个选择是将 RKE 模板与[其他工具](infrastructure.md)一起使用。 + +另一种选择是使用包含节点池配置选项，但不强制执行配置的[集群模板](../../manage-clusters/manage-cluster-templates.md)。 + +## YAML 定制 + +如果将 RKE 模板定义为 YAML 文件，则可以修改此[示例 RKE 模板 YAML](../../../../reference-guides/rke1-template-example-yaml.md)。RKE 模板中的 YAML 使用了 Rancher 在创建 RKE 集群时使用的相同自定义设置。但由于 YAML 要在 Rancher 配置的集群中使用，因此需要将 RKE 模板自定义项嵌套在 YAML 中的 `rancher_kubernetes_engine_config` 参数下。 + +RKE 文档也提供[注释的](https://rancher.com/docs/rke/latest/en/example-yamls/) `cluster.yml` 文件供你参考。 + +有关可用选项的更多信息，请参阅[集群配置](https://rancher.com/docs/rke/latest/en/config-options/)上的 RKE 文档。 + +### 附加组件 + +RKE 模板配置文件的附加组件部分的工作方式与[集群配置文件的附加组件部分](https://rancher.com/docs/rke/latest/en/config-options/add-ons/)相同。 + +用户定义的附加组件指令允许你调用和下拉 Kubernetes 清单或将它们直接内联。如果这些 YAML 清单包括在 RKE 模板中，Rancher 将在集群中部署这些 YAML 文件。 + +你可以使用附加组件执行以下操作： + +- 启动 Kubernetes 集群后，在集群上安装应用 +- 在使用 Kubernetes Daemonset 部署的节点上安装插件 +- 自动设置命名空间、ServiceAccount 或角色绑定 + +RKE 模板配置必须嵌套在 `rancher_kubernetes_engine_config` 参数中。要设置附加组件，在创建模板时单击**以 YAML 文件编辑**。然后使用 `addons` 指令添加清单，或使用 `addons_include` 指令设置哪些 YAML 文件可用于附加组件。有关自定义附加组件的更多信息，请参见[用户自定义附加组件文档](https://rancher.com/docs/rke/latest/en/config-options/add-ons/user-defined-add-ons/)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md new file mode 100644 index 00000000000..aa3c433b873 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md @@ -0,0 +1,11 @@ +--- +title: Kubernetes 集群基础设施 +--- + + + + + +要为具有外部数据库的高可用 K3s Kubernetes 集群设置基础设施，请参见[本页面](ha-k3s-kubernetes-cluster.md)。 + +要为高可用 RKE Kubernetes 集群设置基础设施，请参见[本页面](ha-rke1-kubernetes-cluster.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md new file mode 100644 index 00000000000..f8213ffbbac --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md @@ -0,0 +1,11 @@ +--- +title: "Kubernetes 使用教程" +--- + + + + + +本章节介绍如何安装 Kubernetes 集群，使得 Rancher Server 可以安装在该集群上。 + +Rancher 可以在任何 Kubernetes 集群上运行。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md new file mode 100644 index 00000000000..b1b5457ecf1 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md @@ -0,0 +1,52 @@ +--- +title: 生产就绪集群检查清单 +--- + + + + + +本节将介绍创建生产就绪型 Kubernetes 集群的最佳实践。这个集群可用于运行你的应用和服务。 + +有关集群的要求（包括对 OS/Docker、硬件和网络的要求），请参阅[节点要求](../node-requirements-for-rancher-managed-clusters.md)部分。 + +本文介绍了我们推荐用于所有生产集群的最佳实践的简短列表。 + +如需获取推荐的所有最佳实践的完整列表，请参阅[最佳实践](../../../../reference-guides/best-practices/best-practices.md)部分。 + +### 节点要求 + +* 确保你的节点满足所有[节点要求](../node-requirements-for-rancher-managed-clusters.md)，包括端口要求。 + +### 备份 etcd + +* 启用 etcd 快照。验证是否正在创建快照，并执行灾难恢复方案，从而验证快照是否有效。etcd 是存储集群状态的位置，丢失 etcd 数据意味着丢失集群。因此，请确保为集群配置 etcd 的定期快照，并确保快照也是存储在外部（节点外）的。 + +### 集群架构 + +* 节点应具有以下角色配置之一： + * `etcd` + * `controlplane` + * `etcd` 和 `controlplane` + * `worker`（不应在具有 `etcd` 或 `controlplane` 角色的节点上使用或添加 `worker` 角色） +* 至少拥有三个角色为 `etcd` 的节点，来确保失去一个节点时仍能存活。增加 etcd 节点数量能提高容错率，而将 etcd 分散到不同可用区甚至能获取更好的容错能力。 +* 为两个或更多节点分配 `controlplane` 角色，能实现主组件的高可用性。 +* 为两个或多个节点分配 `worker` 角色，以便在节点故障时重新安排工作负载。 + +有关每个角色的用途的更多信息，请参阅 [Kubernetes 中的节点角色](roles-for-nodes-in-kubernetes.md)。 + +有关每个 Kubernetes 角色的节点数的详细信息，请参阅[推荐架构](../../../../reference-guides/rancher-manager-architecture/architecture-recommendations.md)部分。 + +### Logging 和 Monitoring + +* 为 Kubernetes 组件（系统服务）配置告警/通知程序。 +* 为集群分析和事后剖析配置 Logging。 + +### 可靠性 + +* 在集群上执行负载测试，以验证硬件是否可以支持你的工作负载。 + +### 网络 + +* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`，`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络（网络延迟特别高的情况下除外）中运行。 +* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区，这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 [Kubernetes Cloud Provider](../set-up-cloud-providers/set-up-cloud-providers.md) 资源，请查阅文档以了解限制（即区域存储限制）。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md new file mode 100644 index 00000000000..8cd13712b03 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md @@ -0,0 +1,78 @@ +--- +title: Rancher 中的 Kubernetes 集群设置 +description: 配置 Kubernetes 集群 +--- + + + + + +Rancher 允许你通过 Rancher UI 来创建集群，从而简化了集群的创建流程。Rancher 提供了多种启动集群的选项。你可以选择最适合你的用例的选项。 + +本节默认你已对 Docker 和 Kubernetes 有一定的了解。如果你需要了解 Kubernetes 组件如何协作，请参见 [Kubernetes 概念](../../../reference-guides/kubernetes-concepts.md)页面。 + +有关 Rancher Server 配置集群的方式，以及使用什么工具来创建集群的详细信息，请参阅[产品架构](../../../reference-guides/rancher-manager-architecture/rancher-manager-architecture.md)页面。 + +### 不同类型集群的管理功能 + +下表总结了每一种类型的集群和对应的可编辑的选项和设置： + +import ClusterCapabilitiesTable from '../../../shared-files/\_cluster-capabilities-table.md'; + + + +## 在托管的 Kubernetes 提供商中设置集群 + +在这种情况下，Rancher 不会配置 Kubernetes，因为它是由 Google Kubernetes Engine (GKE)、Amazon Elastic Container Service for Kubernetes 或 Azure Kubernetes Service 等提供商安装的。 + +如果你使用 Kubernetes 提供商，例如 Google GKE，Rancher 将与对应的云 API 集成，允许你从 Rancher UI 为托管集群创建和管理 RBAC。 + +详情请参阅[托管 Kubernetes 集群](set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md)部分。 + +## 使用 Rancher 启动 Kubernetes + +在你自己的节点上配置 Kubernetes 时，Rancher 使用 [Rancher Kubernetes Engine (RKE)](https://rancher.com/docs/rke/latest/en/) 作为库。RKE 是 Rancher 自己的轻量级 Kubernetes 安装程序。 + +在 RKE 集群中，Rancher 管理 Kubernetes 的部署。这些集群可以部署在任何裸机服务器、云提供商或虚拟化平台上。 + +这些节点可以通过 Rancher 的 UI 动态配置，该 UI 调用 [Docker Machine](https://docs.docker.com/machine/) 在各种云提供商上启动节点。 + +如果你已经有一个想要添加到 RKE 集群的节点，你可以通过在节点上运行 Rancher Agent 容器将节点添加到集群中。 + +有关详细信息，请参阅 [RKE 集群](../launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)部分。 + +### 在基础设施提供商中启动 Kubernetes 并配置节点 + +Rancher 可以在 Amazon EC2、DigitalOcean、Azure 或 vSphere 等基础设施提供商中动态配置节点，然后在节点上安装 Kubernetes。 + +使用 Rancher，你可以基于[节点模板](../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md#节点模板)创建节点池。此模板定义了要在云提供商中启动的节点的参数。 + +使用由基础设施提供商托管的节点的一个好处是，如果一个节点与集群失去连接，Rancher 可以自动替换它，从而维护集群配置。 + +Rancher UI 中状态为 Active 的[主机驱动](../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md#主机驱动)决定了可用于创建节点模板的云提供商。 + +如需更多信息，请参阅[基础设施提供商托管的节点](../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)部分。 + +### 在现有自定义节点上启动 Kubernetes + +在设置这种类型的集群时，Rancher 会在现有的[自定义节点](../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)上安装 Kubernetes，从而创建一个自定义集群。 + +你可以使用任何节点，在 Rancher 中创建一个集群。 + +这些节点包括本地裸机服务器、云托管虚拟机或本地虚拟机。 + +## 注册现有集群 + +集群注册功能取代了导入集群的功能。 + +注册 EKS 集群的优点更多。在大多数情况下，注册的 EKS 集群和在 Rancher 中创建的 EKS 集群在 Rancher UI 中的处理方式相同（除了删除）。 + +删除在 Rancher 中创建的 EKS 集群后，该集群将被销毁。删除在 Rancher 中注册的 EKS 集群时，它与 Rancher Server 会断开连接，但它仍然存在。你仍然可以像在 Rancher 中注册之前一样访问它。 + +详情请参见[本页面](register-existing-clusters.md)。 + +## 以编程方式创建集群 + +通过 Rancher 以编程方式部署 Kubernetes 集群的最常见方法是使用 Rancher 2 Terraform Provider。详情请参见[使用 Terraform 创建集群](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster)。 + +你可以使用 Terraform 创建或导入 EKS、GKE、AKS 集群和 RKE 集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-amazon.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-amazon.md new file mode 100644 index 00000000000..fb21c4e82ef --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-amazon.md @@ -0,0 +1,194 @@ +--- +title: 将 Amazon 从树内迁移到树外 +--- + + + + + +Kubernetes 不再在树内维护云提供商。在 Kubernetes v1.27 及更高版本中，树内云提供商已被移除。当你从树内提供商迁移到树外提供商时，Rancher UI 允许你升级到 Kubernetes v1.27。 + +不过，如果你执行的是手动迁移，现有集群必须在迁移后升级到 Kubernetes v1.27 才能继续运行。 + +要从树内云提供商迁移到树外 AWS 云提供商，必须停止现有集群的 kube 控制器管理器，并安装 AWS 云控制器管理器。有许多方法可以做到这一点。有关详情，请参阅有关[外部云控制器管理器](https://cloud-provider-aws.sigs.k8s.io/getting_started/)的 AWS 官方文档。 + +如果可以接受迁移过程中出现一些停机，请按照说明[设置外部云提供商](../set-up-cloud-providers/amazon.md#using-the-out-of-tree-aws-cloud-provider)。这些说明概述了如何为新配置的集群配置树外云提供商。在设置过程中，会有一些停机，因为从旧云提供商停止运行到新云提供商开始运行之间会有一段时间的间隔。 + +如果您的设置不能容忍任何控制平面停机，则必须启用领导者迁移。这有助于从 kube 控制器管理器中的控制器顺利过渡到云控制器管理器中的对应控制器。有关详细信息，请参阅 AWS 官方文档[使用领导者迁移](https://cloud-provider-aws.sigs.k8s.io/getting_started/)。 + +:::note Important: +Kubernetes [云控制器迁移文档](https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/#before-you-begin)指出，可以使用相同的 Kubernetes 版本进行迁移，但假设迁移是 Kubernetes 升级的一部分。请参考有关[迁移到要使用的云控制器管理器](https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/)的 Kubernetes 文档，了解迁移前是否需要自定义设置。确认[迁移配置值](https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/#default-configuration)。如果您的云提供商提供 Node IPAM 控制器的实现，您还需要迁移 [IPAM 控制器](https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/#node-ipam-controller-migration)。 +::: + + + + +1. 更新集群配置，启用领导者迁移： + +```yaml +spec: + rkeConfig: + machineSelectorConfig: + - config: + kube-controller-manager-arg: + - enable-leader-migration + machineLabelSelector: + matchExpressions: + - key: rke.cattle.io/control-plane-role + operator: In + values: + - "true" +``` + +请注意，云提供商在此步骤中仍是 `aws` + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + cloud-provider-name: aws +``` + +2. 标记控制平面节点不可调度，以便 AWS 云控制器 pod 仅在升级到外部云提供商后才在节点上运行： + +```shell +kubectl cordon -l "node-role.kubernetes.io/control-plane=true" +``` + +3. 要安装启用了领导者迁移的 AWS 云控制器管理器，请遵循[部署云控制器管理器 Chart](../set-up-cloud-providers/amazon.md#using-the-out-of-tree-aws-cloud-provider) 的步骤 1-3。从 Kubernetes 1.22 起，kube-controller-manager 将使用默认配置，该配置将满足 controller-to-manager 的迁移。更新 `spec.rkeConfig.additionalManifest` 下 `aws-cloud-controller-manager` 的容器参数，以启用领导者迁移： + +```shell +- '--enable-leader-migration=true' +``` + +4. 安装 Chart 并确认 Daemonset `aws-cloud-controller-manager` 已成功部署： + +```shell +kubectl rollout status daemonset -n kube-system aws-cloud-controller-manager +``` + +5. 更新配置集群以更改云提供商，并从 kube 控制器中移除领导者迁移 参数。如果升级 Kubernetes 版本，也要在集群 YAML 文件的 `spec.kubernetesVersion` 部分设置 Kubernetes 版本 + +:::note Important + +如果不依赖 rke2 监管程序正确设置提供商 ID，则只移除 `cloud-provider-name: aws`。 + +::: + +如果不想在集群中启用 `enable-leader-migration` 功能，请将其移除： + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + cloud-provider-name: external +``` + +移除 `enable-leader-migration`： + +```yaml +spec: + rkeConfig: + machineSelectorConfig: + - config: + kube-controller-manager-arg: + - enable-leader-migration + machineLabelSelector: + matchExpressions: + - key: rke.cattle.io/control-plane-role + operator: In + values: + - "true" +``` + +:::tip + +您也可以在升级后禁用领导者迁移，因为由于只有一个云控制器管理器，不再需要领导者迁移，可以将其移除。升级 Chart 并从容器参数中删除以下部分： + +```yaml +- --enable-leader-migration=true +``` + +::: + +使用以下命令验证云控制器管理器更新是否成功上线： + +```shell +kubectl rollout status daemonset -n kube-system aws-cloud-controller-manager +``` + +6. 云提供商负责设置节点的 ProviderID。检查是否所有节点都用 ProviderID 进行了初始化： + +```shell +kubectl describe nodes | grep "ProviderID" +``` + + + + + +1. 更新集群配置，在 `cluster.yml` 中启用领导者迁移： + +```yaml +services: + kube-controller: + extra_args: + enable-leader-migration: "true" +``` + +请注意，云提供商在此步骤中仍是 `aws` + +```yaml +cloud_provider: + name: aws +``` + +2. 标记控制平面节点不可调度，以便 AWS 云控制器 pod 仅在升级到外部云提供商后才在节点上运行： + +```shell +kubectl cordon -l "node-role.kubernetes.io/controlplane=true" +``` + +3. 要安装 AWS 云控制器管理器，必须启用领导者迁移，并遵循在新集群上安装 AWS 时的相同步骤。要启用领导者迁移，请在步骤 7 中的容器参数中添加以下内容，同时[按照步骤安装 Chart](../set-up-cloud-providers/amazon.md#helm-chart-installation-from-ui)： + +```yaml +- "--enable-leader-migration=true" +``` + +4. 确认 Chart 已安装，但由于控制面板节点不可调度，新的 pod 尚未运行。在下一步更新集群后，RKE 将升级并允许对每个节点的调度，并调度 `aws-controller-manager` pod。 + +5. 更新 `cluster.yml`以更改云提供商，并从 kube-controller 中移除领导者迁移参数。 + +选择 **External Amazon (out-of-tree)** 可设置 `--cloud-provider=external`，并启用 `useInstanceMetadataHostname`。节点驱动集群和自定义集群必须启用 `useInstanceMetadataHostname`，否则无法通过 `--node-name`提供自定义节点名称。启用 `useInstanceMetadataHostname` 会查询 ec2 元数据服务，并为 `kubelet` 和 `kube-proxy`将 `/hostname` 设置为 `hostname-override` ： + +```yaml +rancher_kubernetes_engine_config: + cloud_provider: + name: external-aws + useInstanceMetadataHostname: true/false +``` + +如果不想在集群中启用 `enable-leader-migration`，请移除它： + +```yaml +services: + kube-controller: + extra_args: + enable-leader-migration: "true" +``` + +:::tip +完成迁移后，还可以禁用领导者迁移。升级 Chart 并从容器参数中删除以下部分： + +```yaml +- --enable-leader-migration=true +``` + +::: + +6. 如果要升级集群的 Kubernetes 版本，也要设置 Kubernetes 版本。 + +7. 更新集群。现在，`aws-cloud-controller-manager` pod 应该已经运行。 + + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md new file mode 100644 index 00000000000..3649e928567 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md @@ -0,0 +1,111 @@ +--- +title: 将 vSphere 从树内迁移到树外 +--- + + + + + +Kubernetes 不再在树内维护云提供商。vSphere 有一个树外云提供商，可以通过安装 vSphere 云提供商和云存储插件来使用。 + +本页面介绍如何从树内 vSphere 云提供商迁移到树外，并在迁移后管理现有虚拟机。 + +它遵循官方 [vSphere 迁移文档](https://vsphere-csi-driver.sigs.k8s.io/features/vsphere_csi_migration.html)中提供的步骤，并提供在 Rancher 中执行的步骤。 + +### Cloud-config 格式限制 + +由于 vSphere Cloud Storage Interface (CSI) 中的一个现有错误，使用以下 cloud-config 格式配置的现有卷将无法迁移。 + +如果 cloud-config 中的数据存储和资源池路径采用这种格式，vsphere CSI 驱动程序将无法识别它： + +```yaml +default-datastore: /datastore/ +resourcepool-path: "/host//Resources/" +``` + +使用树内提供商以下列格式配置的卷将正确迁移： + +```yaml +default-datastore: +resourcepool-path: "/Resources/" +``` + +上游 bug: https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/628 + +Rancher 问题跟踪此 bug: https://github.com/rancher/rancher/issues/31105 + +## 前提条件 + +- vSphere CSI 迁移需要 vSphere 7.0u1。为了能够管理现有树内 vSphere 卷，请将 vSphere 升级到 7.0u1。 +- Kubernetes 版本必须为 1.19 或更高。 + +## 迁移 + +### 1. 安装 CPI 插件 + +安装 CPI 之前，我们需要用 `node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule` 标记所有节点。 + +可以通过运行以下命令来完成： + +``` +curl -O https://raw.githubusercontent.com/rancher/helm3-charts/56b622f519728378abeddfe95074f1b87ab73b1e/charts/vsphere-cpi/taints.sh +``` + +或: + +``` +wget https://raw.githubusercontent.com/rancher/helm3-charts/56b622f519728378abeddfe95074f1b87ab73b1e/charts/vsphere-cpi/taints.sh +chmod +x taints.sh +./taints.sh +``` + +运行脚本并标记所有节点后，启动 Helm vSphere CPI Chart。 + +1. 点击 **☰ > Cluster Management**. +1. 转到将要安装 vSphere CPI Chart 的集群，然后点击 **Explore**. +1. 点击 **Apps > Charts**. +1. 点击 **vSphere CPI**. +1. 点击 **Install**. +1. 填写所需的 vCenter 详细信息，然后单击 **Install**. + +vSphere CPI 会使用 ProviderID 初始化所有节点，这是 vSphere CSI 驱动程序所需要的。 + +使用以下命令检查是否用 ProviderID 初始化了所有节点： + +``` +kubectl describe nodes | grep "ProviderID" +``` + +### 2. 安装 CSI 驱动程序 + +1. 点击 **☰ > Cluster Management**. +1. 转到将要安装 vSphere CSI Chart 的集群，点击 **Explore**. +1. 点击 **Apps > Charts**. +1. 点击 **vSphere CSI**. +1. 点击 **Install**. +1. 填写所需的 vCenter 详细信息，然后单击 **Install**. +1. 选中 **Customize Helm options before install**，然后单击 **Next**. +1. 在 **Features** 选项卡上， 选中 **Enable CSI Migration**. +1. （可选）转到 **Storage** 选项卡设置数据存储。此 Chart 会创建一个以 `csi.vsphere.vmware.com` 作为供应商的 StorageClass。在创建此 StorageClass 时，您可以提供用于 CSI 卷配置的数据存储的 URL。数据存储 URL 可在 vSphere 客户端中通过选择数据存储并转到摘要选项卡找到。填写 StorageClass 的详细信息。 +1. 点击 **Install**. + +### 3. 编辑集群以启用 CSI 迁移功能标志 + +1. 编辑集群时，如果 Kubernetes 版本低于 1.19，请从 **Kubernetes Version** 下拉菜单中选择 1.19 或更高版本。 +2. 要启用功能标志，请单击 "Edit as YAML", 然后在 kube-controller 和 kubelet 下添加以下内容: + + ```yaml + extra_args: + feature-gates: "CSIMigration=true,CSIMigrationvSphere=true" + ``` + +### 4. 清空工作节点 + +在升级过程中，必须先清空工作节点，然后再更改kubelet和kube-controller-manager参数。 + +1. 点击 **☰ > Cluster Management**。 +1. 转到要清空工作节点的集群，点击 **⋮ > Edit Config**。 +1. 在 **Advanced Options** 部分，将 **Maximum Worker Nodes Unavailable** 字段设为 1。 +1. 要在升级期间清空节点，请选择 **Drain Nodes > Yes**. +1. 将 **Force** 和 **Delete Local Data** 设置为 **true**. +1. 点击 **Save** 升级集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md new file mode 100644 index 00000000000..7508802ff08 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md @@ -0,0 +1,47 @@ +--- +title: 设置 Cloud Provider +--- + + + + + +_cloud provider_ 是 Kubernetes 中的一个模块，它提供了一个用于管理节点、负载均衡器和网络路由的接口。 + +在 Rancher 中设置 cloud provider 时，如果你使用的云提供商支持自动化，Rancher Server 可以在启动 Kubernetes 定义时自动配置新节点、负载均衡器或持久存储设备。 + +如果你配置的节点云提供商集群不满足先决条件，集群将无法正确配置。 + +**Cloud Provider** 选项默认设置为 `None`。 + +可以启用的云提供商包括： + +* Amazon +* Azure +* GCE (Google Compute Engine) +* vSphere + +### 设置 Amazon 云提供商 + +有关启用 Amazon 云提供商的详细信息，请参阅[此页面](amazon.md)。 + +### 设置 Azure 云提供商 + +有关启用 Azure 云提供商的详细信息，请参阅[此页面](azure.md)。 + +### 设置 GCE 云提供商 + +有关启用 Google Compute Engine 云提供商的详细信息，请参阅[此页面](google-compute-engine.md)。 + +### 设置 vSphere 云提供商 + +有关启用 vSphere 云提供商的详细信息，请参阅[树内 vSphere 配置](configure-in-tree-vsphere.md) 和 [树外 vSphere 配置](configure-out-of-tree-vsphere.md)。 + +### 设置自定义云提供商 + +如果您想配置其他 Kubernetes 云提供商，则可使用 `自定义` 云提供商。 + +关于自定义云提供商选项，您可以参考 [RKE 文档](https://rancher.com/docs/rke/latest/en/config-options/cloud-providers/)，了解如何为你的云提供商编辑 yaml 文件。特定云提供商的详细配置说明如下： + +- [vSphere](https://rke.docs.rancher.com/config-options/cloud-providers/vsphere) +- [OpenStack](https://rancher.com/docs/rke/latest/en/config-options/cloud-providers/openstack/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md new file mode 100644 index 00000000000..51662df0829 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md @@ -0,0 +1,283 @@ +--- +title: 在 Windows 集群上启动 Kubernetes +--- + + + + + +使用 Rancher 配置[自定义集群](../../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)时，Rancher 通过 RKE（Rancher Kubernetes Engine）在现有节点上安装 Kubernetes。 + +在使用 Rancher 配置的 Windows 集群中，集群必须同时包含 Linux 和 Windows 节点。Kubernetes controlplane 只能运行在 Linux 节点上，Windows 节点只能有 Worker 角色。Windows 节点只能用于部署工作负载。 + +Windows 集群的其他要求如下： + +- 只有在创建集群时启用了 Windows 支持的集群才能添加 Windows 节点。无法为现有集群启用 Windows 支持。 +- 需要 Kubernetes 1.15+。 +- 必须使用 Flannel 网络提供商。 +- Windows 节点必须有 50 GB 的磁盘空间。 + +有关完整的要求列表，请参阅[本节](#windows-集群的要求)。 + +有关支持 Windows 的 Kubernetes 功能摘要，请参阅[在 Windows 中使用 Kubernetes 支持的功能和限制](https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#supported-functionality-and-limitations)的 Kubernetes 文档，或[在 Kubernetes 中调度 Windows 容器的指南](https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-containers/)。 + +### RKE2 Windows + +RKE2 配置功能还包括在 Windows 集群上安装 RKE2。RKE2 的 Windows 功能包括： + +- 由 containerd 提供支持的使用 RKE2 的 Windows 容器 +- 直接从 Rancher UI 配置 Windows RKE2 自定义集群 +- 用于 Windows RKE2 自定义集群的 Calico CNI +- 技术预览包含了 Windows Server 的 SAC 版本（2004 和 20H2） + +要使 Windows 支持 RKE2 自定义集群，请选择 Calico 作为 CNI。 + +:::note + +默认情况下，Rancher 允许 Windows 工作负载 pod 部署在 Windows 和 Linux Worker 节点上。在 RKE2 中创建混合集群时，你必须编辑 Chart 中的 `nodeSelector`，从而将 Pod 放置到兼容的 Windows 节点上。有关如何使用 `nodeSelector` 将 pod 分配给节点的更多信息，请参阅 [Kubernetes 文档](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)。 +::: + +- Kubernetes v1.24.1 及更高版本支持 Windows RKE2 中的 HostProcess 容器。有关详细信息，请参阅[上游文档](https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/)。 + +## Windows 集群的要求 + +网络、操作系统和 Docker 的一般节点要求与 [Rancher 安装](../../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md) 的节点要求相同。 + +### 操作系统和 Docker 要求 + +我们对 Windows Server 和 Windows 容器的支持与 LTSC（长期服务渠道）和 SAC（半年渠道）的 Microsoft 官方生命周期相匹配。 + +有关 Windows Server 的支持生命周期的日期，请参阅 [Microsoft 文档](https://docs.microsoft.com/en-us/windows-server/get-started/windows-server-release-info)。 + +### Kubernetes 版本 + +需要 Kubernetes v1.15+。 + +如果你在 Windows Server 20H2 Standard Core 上使用 Kubernetes v1.21，则必须在节点上安装补丁“2019-08 Servicing Stack Update for Windows Server”。 + +### 节点要求 + +集群中的主机至少需要： + +- 2 核 CPU +- 5 GB 内存 +- 50 GB 磁盘空间 + +Rancher 不会配置不满足要求的节点。 + +### 网络要求 + +在配置新集群之前，请确保你已经在接收入站网络流量的设备上安装了 Rancher。这是集群节点与 Rancher 通信所必需的。如果你尚未安装 Rancher，请在继续阅读本指南之前先参阅[安装文档](../../../../getting-started/installation-and-upgrade/installation-and-upgrade.md)进行安装。。 + +Rancher 仅支持使用 Flannel 作为网络提供商的 Windows。 + +有两个网络选项：[**Host Gateway (L2bridge)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) 和 [**VXLAN (Overlay)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan)。默认选项是 **VXLAN (Overlay)** 模式。 + +对于 **Host Gateway (L2bridge)** 网络，最好为所有节点使用相同的第 2 层网络。否则，你需要为它们配置路由规则。有关详细信息，请参阅[配置云托管 VM 路由的文档](network-requirements-for-host-gateway.md#云托管虚拟机的路由配置)。如果你使用的是 Amazon EC2、Google GCE 或 Azure 虚拟机，你需要[禁用私有 IP 地址检查](network-requirements-for-host-gateway.md#禁用私有-ip-地址检查)。 + +对于 **VXLAN (Overlay)** 网络，你必须安装 [KB4489899](https://support.microsoft.com/en-us/help/4489899) 补丁。大多数云托管的 VM 已经具有此修补程序。 + +如果你在为 AWS 虚拟私有云配置 DHCP 选项集，请注意，你只能在 `domain-name` 选项字段中指定一个域名。详情请参见 [DHCP 选项文档](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html)： + +:::note + +一些 Linux 操作系统支持以空格分隔的多个域名。但是，其他 Linux 操作系统和 Windows 将该值视为单个域名，从而导致意外错误。如果你的 DHCP 选项集与具有多个操作系统实例的 VPC 相关联，请仅指定一个域名。 + +::: + +### 带有 ESXi 6.7u2 及更高版本的 vSphere 上的 Rancher + +如果你在带有 ESXi 6.7u2 或更高版本的 VMware vSphere 上使用 Rancher，并使用 Red Hat Enterprise Linux 8.3、CentOS 8.3 或 SUSE Enterprise Linux 15 SP2 或更高版本，你需要禁用 `vmxnet3` 虚拟网络适配器硬件卸载功能。否则，不同集群节点上的 pod 之间的所有网络连接会因为超时错误而失败。从 Windows pod 到在 Linux 节点上运行的关键服务（例如 CoreDNS）的所有连接也将失败。外部连接也可能失败。出现这个问题的原因是 Linux 发行版在 `vmxnet3` 中启用了硬件卸载功能，而且 `vmxnet3` 硬件卸载功能中存在一个会丢弃客户覆盖流量的数据包的 bug。要解决此问题，必须禁用 `vmxnet3` 硬件卸载功能。此设置不会在重启后继续生效，因此需要在每次启动时禁用。推荐的做法是在 `/etc/systemd/system/disable_hw_offloading.service` 中创建一个 systemd 单元文件，这会在启动时禁用 `vmxnet3` 硬件卸载功能。禁用 `vmxnet3` 硬件卸载功能的示例 systemd 单元文件如下所示。注意，`` 必须自定义为主机的 `vmxnet3` 网络接口，如 `ens192`： + +``` +[Unit] +Description=Disable vmxnet3 hardware offloading feature + +[Service] +Type=oneshot +ExecStart=ethtool -K tx-udp_tnl-segmentation off +ExecStart=ethtool -K tx-udp_tnl-csum-segmentation off +StandardOutput=journal + +[Install] +WantedBy=multi-user.target +``` + +然后在 systemd 单元文件上设置适当的权限： + +``` +chmod 0644 /etc/systemd/system/disable_hw_offloading.service +``` + +最后，启用 systemd 服务： + +``` +systemctl enable disable_hw_offloading.service +``` + +### 架构要求 + +Kubernetes 集群管理节点（`etcd` 和 `controlplane`）必须运行在 Linux 节点上。 + +部署工作负载的 `worker` 节点通常是 Windows 节点，但必须至少有一个 `worker` 节点运行在 Linux 上，才能按顺序运行 Rancher Cluster Agent、DNS、Metrics Server 和 Ingress 相关容器。 + +#### 推荐架构 + +我们推荐下表中列出的三节点架构，但你始终可以添加额外的 Linux 和 Windows worker 节点来扩展集群，从而实现冗余： + +| 节点 | 操作系统 | Kubernetes 集群角色 | 用途 | +| ------ | --------------------------------------------- | --------------------------- | ----------------------------------------------------------- | +| 节点 1 | Linux (推荐 Ubuntu Server 18.04) | Control plane, etcd, worker | 管理 Kubernetes 集群 | +| 节点 2 | Linux (推荐 Ubuntu Server 18.04) | Worker | 支持集群的 Rancher Cluster Agent、Metrics Server、DNS 和 Ingress | +| 节点 3 | Windows（Windows Server 核心版本 1809 或更高版本） | Worker | 运行 Windows 容器 | + +### 容器要求 + +Windows 要求容器的版本必须与部署容器的 Windows Server 的版本一致。因此，你必须在 Windows Server 核心版本 1809 或更高版本上构建容器。如果你已经使用早期的 Windows Server 核心版本构建了容器，则必须使用 Windows Server 核心版本 1809 或更高版本重新构建容器。 + +### 云提供商要求 + +如果你在集群中设置了 Kubernetes 云提供商，则需要进行一些额外的操作。如果你想使用云提供商的功能，例如为集群自动配置存储、负载均衡器或其他基础设施，你可能需要设置云提供商。有关如何配置满足条件的云提供商集群节点，请参阅[此页面](../set-up-cloud-providers/set-up-cloud-providers.md)。 + +如果你的云提供商是 GCE（Google Compute Engine），则必须执行以下操作： + +- 按照[步骤](../set-up-cloud-providers/google-compute-engine.md)在`cluster.yml` 中启用 GCE 云提供商。 +- 在 Rancher 中配置集群时，在 Rancher UI 中选择**自定义云提供商**作为云提供商。 + +## 教程：如何创建支持 Windows 的集群 + +本教程描述了如何使用[推荐架构](#推荐架构)中的三个节点创建由 Rancher 配置的集群。 + +在现有节点上使用 Rancher 配置集群时，你需要在每个节点上安装 [Rancher Agent](../../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/rancher-agent-options.md) 来将节点添加到集群中。在 Rancher UI 中创建或编辑集群时，你会看到一个**自定义节点运行命令**，你可以在每台服务器上运行该命令，从而将服务器添加到集群中。 + +要设置支持 Windows 节点和容器的集群，你需要完成以下任务： + +### 1. 配置主机 + +要在具有 Windows 支持的现有节点上配置集群，请准备好你的主机。 + +主机可以是： + +- 云托管的虚拟机 +- 虚拟化集群中的虚拟机 +- 裸金属服务器 + +你将配置三个节点： + +- 一个 Linux 节点，用于管理 Kubernetes controlplane 并存储你的 `etcd`。 +- 第二个 Linux 节点，它将作为 worker 节点。 +- Windows 节点，它将作为 worker 节点运行 Windows 容器。 + +| 节点 | 操作系统 | +| ------ | --------------------------------------------- | +| 节点 1 | Linux (推荐 Ubuntu Server 18.04) | +| 节点 2 | Linux (推荐 Ubuntu Server 18.04) | +| 节点 3 | Windows（Windows Server 核心版本 1809 或更高版本） | + +如果你的节点托管在**云提供商**上，并且你需要自动化支持（例如负载均衡器或持久存储设备），你的节点还需要满足额外的配置要求。详情请参见[选择云提供商](../set-up-cloud-providers/set-up-cloud-providers.md)。 + +### 2. 在现有节点上创建集群 + +在现有节点上创建 Windows 集群的说明与一般[创建自定义集群的说明](../../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)非常相似，但有一些特定于 Windows 的要求。 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面上，单击**创建**。 +1. 单击**自定义**。 +1. 在**集群名称**字段中输入集群的名称。 +1. 在 **Kubernetes 版本**下拉菜单中，选择 v1.19 或更高版本。 +1. 在**网络提供商**字段中，选择 **Flannel**。 +1. 在 **Windows 支持**中，单击**启用**。 +1. 可选：启用 Windows 支持后，你将能够选择 Flannel 后端模式。有两个网络选项：[**Host Gateway (L2bridge)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) 和 [**VXLAN (Overlay)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan)。 默认选项是 **VXLAN (Overlay)** 模式。 +1. 点击**下一步**。 + +:::note 重要提示： + +对于 Host Gateway (L2bridge) 网络，最好为所有节点使用相同的第 2 层网络。否则，你需要为它们配置路由规则。有关详细信息，请参阅[配置云托管 VM 路由的文档](network-requirements-for-host-gateway.md#云托管虚拟机的路由配置)。如果你使用的是 Amazon EC2、Google GCE 或 Azure 虚拟机，你需要[禁用私有 IP 地址检查](network-requirements-for-host-gateway.md#禁用私有-ip-地址检查)。 + +::: + +### 3. 将节点添加到集群 + +本节介绍如何将 Linux 和 Worker 节点注册到集群。你将在每个节点上运行一个命令，该命令将安装 Rancher Agent 并允许 Rancher 管理每个节点。 + +#### 添加 Linux master 节点 + +在本节中，你需要在 Rancher UI 上填写表单以获取自定义命令，从而在 Linux master 节点上安装 Rancher Agent。然后，复制该命令并在 Linux master 节点上运行命令，从而在集群中注册该节点。 + +集群中的第一个节点应该是具有 **controlplane** 和 **etcd** 角色的 Linux 主机。至少必须为此节点启用这两个角色，并且必须先将此节点添加到集群中，然后才能添加 Windows 主机。 + +1. 在**节点操作系统**中，单击 **Linux**。 +1. 在**节点角色**中，至少选择 **etcd** 和 **controlplane**。推荐选择所有的三个角色。 +1. 可选：如果点击**显示高级选项**，你可以自定义 [Rancher Agent](../../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/rancher-agent-options.md)和[节点标签](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)的设置。 +1. 将屏幕上显示的命令复制到剪贴板。 +1. SSH 到你的 Linux 主机，然后运行复制到剪贴板的命令。 +1. 完成配置 Linux 节点后，选择**完成**。 + +**结果**： + +你已创建集群，集群的状态是**配置中**。Rancher 已在你的集群中。 + +当集群状态变为 **Active** 后，你可访问集群。 + +**Active** 状态的集群会分配到两个项目： + +- `Default`：包含 `default` 命名空间 +- `System`：包含 `cattle-system`，`ingress-nginx`，`kube-public` 和 `kube-system` 命名空间。 + + +节点可能需要几分钟才能注册到集群中。 + +#### 添加 Linux Worker 节点 + +在本节中，我们通过运行命令将 Linux Worker 节点注册到集群中。 + +在初始配置集群之后，你的集群只有一个 Linux 主机。接下来，我们添加另一个 Linux `worker` 主机，用于支持集群的 _Rancher Cluster Agent_、_Metrics Server_、_DNS_ 和 _Ingress_。 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 转到你创建的集群，然后单击 **⋮ > 编辑配置**。 +1. 向下滚动到**节点操作系统**。选择 **Linux**。 +1. 在**自定义节点运行命令**中，转到**节点选项**并选择 **Worker** 角色。 +1. 将屏幕上显示的命令复制到剪贴板。 +1. 使用远程终端连接登录到你的 Linux 主机。粘贴剪贴板的命令并运行。 +1. 在 **Rancher**中，单击**保存**。 + +**结果**：**Worker** 角色已安装在你的 Linux 主机上，并且节点会向 Rancher 注册。节点可能需要几分钟才能注册到集群中。 + +:::note + +Linux Worker 节点上的污点 + +以下污点将添加集群中的 Linux Worker 节点中。将此污点添加到 Linux Worker 节点后，添加到 Windows 集群的任何工作负载都将自动调度到 Windows Worker 节点。如果想将工作负载专门调度到 Linux Worker 节点上，则需要为这些工作负载添加容忍度。 + +| 污点键 | 污点值 | 污点效果 | +| -------------- | ---------- | ------------ | +| `cattle.io/os` | `linux` | `NoSchedule` | + +::: + +#### 添加 Windows Worker 节点 + +在本节中，我们通过运行命令将 Windows Worker 节点注册到集群中。 + +你可以通过编辑集群并选择 **Windows** 选项，从而将 Windows 主机添加到集群中。 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 转到你创建的集群，然后单击 **⋮ > 编辑配置**。 +1. 向下滚动到**节点操作系统**。选择 **Windows**。注意：你将看到 **worker** 角色是唯一可用的角色。 +1. 将屏幕上显示的命令复制到剪贴板。 +1. 使用你喜欢的工具（例如 [Microsoft 远程桌面](https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/remote-desktop-clients)）登录到 Windows 主机。在 **Command Prompt (CMD)** 中运行复制到剪贴板的命令。 +1. 在 Rancher 中，单击**保存**。 +1. 可选：如果要向集群添加更多 Windows 节点，请重复这些操作。 + +**结果**：**Worker** 角色已安装在你的 Windows 主机上，并且节点会向 Rancher 注册。节点可能需要几分钟才能注册到集群中。你现在已拥有一个 Windows Kubernetes 集群。 + +### 可选的后续步骤 + +创建集群后，你可以通过 Rancher UI 访问集群。最佳实践建议你设置以下访问集群的备用方式： + +- **通过 kubectl CLI 访问你的集群**：按照[这些步骤](../../manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#在工作站使用-kubectl-访问集群)在你的工作站上使用 kubectl 访问集群。在这种情况下，你将通过 Rancher Server 的身份验证代理进行身份验证，然后 Rancher 会让你连接到下游集群。此方法允许你在没有 Rancher UI 的情况下管理集群。 +- **通过 kubectl CLI 使用授权的集群端点访问你的集群**：按照[这些步骤](../../manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#直接使用下游集群进行身份验证)直接使用 kubectl 访问集群，而无需通过 Rancher Server 进行身份验证。我们建议设置此替代方法来访问集群，以便在无法连接到 Rancher 时访问集群。 + +## Azure 中存储类的配置 + +如果你的节点使用 Azure VM，则可以使用 [Azure 文件](https://docs.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv)作为集群的存储类（StorageClass）。详情请参见[此部分](azure-storageclass-configuration.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md new file mode 100644 index 00000000000..ba82844f0f4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md @@ -0,0 +1,65 @@ +--- +title: 访问集群 +--- + + + + + +本节介绍可以用来访问 Rancher 管理的集群的工具。 + +有关如何授予用户访问集群的权限的信息，请参阅[将用户添加到集群](add-users-to-clusters.md)部分。 + +有关 RBAC 的更多信息，请参阅[本节](../../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md)。 + +有关如何设置认证的信息，请参阅[本节](../../authentication-permissions-and-global-configuration/authentication-config/authentication-config.md)。 + +## Rancher UI 中的集群 + +我们提供了多种通过 Rancher UI 查看和管理集群的方法。 + +### 集群页面 + +从 **☰** 菜单访问**集群**页面： + +1. 单击 **☰**。 +1. 选择**集群管理**。 + +你还可以通过单击 Rancher UI **主页**集群表格上方的**管理**按钮来访问**集群**页面。 + +在**集群**页面上，选择每行末尾的 **⁝** 以查看包含以下选项的子菜单： + +* [Kubectl Shell](use-kubectl-and-kubeconfig.md) +* 下载 KubeConfig +* 将 KubeConfig 复制到剪切板 +* 编辑配置 +* 查看 YAML +* 下载 YAML + +### 集群仪表板 + +在**集群**页面上，选择每行末尾的**浏览**按钮查看该集群的**集群仪表板**。你还可以通过单击表中集群的名称，然后单击**集群**页面上的**浏览**按钮来查看仪表板。 + +也可以通过单击集群名称从 Rancher UI **主页**访问**集群仪表板**。 + +你还可以从顶部导航栏中的 **☰** 访问**集群仪表板**： + +1. 单击 **☰**。 +1. 从**浏览集群**菜单中选择集群的名称。 + +**集群仪表板**列出了集群相关的信息，例如节点数量、内存使用情况、事件和资源。 + +## kubectl + +你可以使用 Kubernetes 命令行工具 [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)，来管理你的集群。使用 kubectl 有两种选择： + +- **Rancher kubectl shell**：通过启动 Rancher UI 中可用的 kubectl shell 与集群交互。此选项不需要你进行任何配置操作。有关详细信息，请参阅[使用 kubectl Shell 访问集群](use-kubectl-and-kubeconfig.md)。 +- **终端远程连接**：你也可以通过在本地桌面上安装 [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)，然后将集群的 kubeconfig 文件复制到本地 `~/.kube/config` 目录来与集群交互。有关更多信息，请参阅[使用 kubectl 和 kubeconfig 文件访问集群](use-kubectl-and-kubeconfig.md)。 + +## Rancher CLI + +你可以下载 Rancher 自己的命令行工具 [Rancher CLI](../../../../reference-guides/cli-with-rancher/cli-with-rancher.md) 来控制你的集群。这个 CLI 工具可以直接与不同的集群和项目进行交互，或者向它们传递 `kubectl` 命令。 + +## Rancher API + +最后，你可以通过 Rancher API 与集群进行交互。在使用 API 之前，你必须先获取 [API 密钥](../../../../reference-guides/user-settings/api-keys.md)。要查看 API 对象的不同资源字段和操作，请打开 API UI（API UI 可以通过单击 Rancher UI 对象的**在 API 中查看**访问）。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md new file mode 100644 index 00000000000..31c7c4ca4fd --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -0,0 +1,79 @@ +--- +title: "Kubernetes 持久存储： 卷和存储类" +description: "了解在 Kubernetes 中创建持久存储的两种方法：持久卷和存储类" +--- + + + + + +在部署需要保​​留数据的应用时，你需要创建持久存储。持久存储允许你在运行应用的 pod 之外存储应用数据。即使运行应用的 pod 发生故障，这种存储方式也能让你保留应用数据。 + +本文假设你已了解 Kubernetes 的持久卷、持久卷声明和存储类的概念。如需更多信息，请参阅[存储的工作原理](manage-persistent-storage/about-persistent-storage.md)部分。 + +### 先决条件 + +设置持久存储需要`管理卷`的[角色](../../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色参考)。 + +如果你要为云集群配置存储，则存储和集群主机必须使用相同的云提供商。 + +要使用 Rancher 配置新存储，则必须启用云提供商。有关启用云提供商的详细信息，请参阅[此页面](../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md)。 + +如果要将现有的持久存储连接到集群，则不需要启用云提供商。 + +### 设置现有存储 + +设置现有存储的总体流程如下： + +1. 设置你的持久存储。可以是云存储或你自己的存储。 +2. 添加引用持久存储的持久卷 (PV)。 +3. 添加引用 PV 的持久卷声明 (PVC)。 +4. 将 PVC 挂载为工作负载中的卷。 + +有关详细信息和先决条件，请参阅[此页面](manage-persistent-storage/set-up-existing-storage.md)。 + +### 在 Rancher 中动态配置新存储 + +配置新存储的总体流程如下： + +1. 添加一个 StorageClass 并将它配置为使用你的存储提供商。StorageClass 可以引用云存储或你自己的存储。 +2. 添加引用存储类的持久卷声明 (PVC)。 +3. 将 PVC 挂载为工作负载的卷。 + +有关详细信息和先决条件，请参阅[此页面](manage-persistent-storage/dynamically-provision-new-storage.md)。 + +### Longhorn 存储 + +[Longhorn](https://longhorn.io/) 是一个轻量级、可靠、易用的 Kubernetes 分布式块存储系统。 + +Longhorn 是免费的开源软件。Longhorn 最初由 Rancher Labs 开发，现在正在作为云原生计算基金会的沙盒项目进行开发。它可以通过 Helm、kubectl 或 Rancher UI 安装在任何 Kubernetes 集群上。 + +如果你有块存储池，Longhorn 可以帮助你为 Kubernetes 集群提供持久存储，而无需依赖云提供商。有关 Longhorn 功能的更多信息，请参阅[文档](https://longhorn.io/docs/latest/what-is-longhorn/)。 + +Rancher v2.5 简化了在 Rancher 管理的集群上安装 Longhorn 的过程。详情请参见[本页面](../../../../integrations-in-rancher/longhorn/longhorn.md)。 + +### 配置存储示例 + +我们提供了如何使用 [NFS](../provisioning-storage-examples/nfs-storage.md)， [vSphere](../provisioning-storage-examples/vsphere-storage.md)，和 [Amazon 的 EBS](../provisioning-storage-examples/persistent-storage-in-amazon-ebs.md) 来配置存储的示例。 + +### GlusterFS 卷 + +在将数据存储在 GlusterFS 卷上的集群中，你可能会遇到重启 `kubelet` 后 pod 无法挂载卷的问题。有关避免此情况发生的详细信息，请参阅[此页面](manage-persistent-storage/about-glusterfs-volumes.md)。 + +### iSCSI 卷 + +在将数据存储在 iSCSI 卷上的 [Rancher 启动的 Kubernetes 集群](../../launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)中，你可能会遇到 kubelet 无法自动连接 iSCSI 卷的问题。有关解决此问题的详细信息，请参阅[此页面](manage-persistent-storage/install-iscsi-volumes.md)。 + +### hostPath 卷 + +在创建 hostPath 卷之前，你需要在集群配置中设置 [extra_bind](https://rancher.com/docs/rke/latest/en/config-options/services/services-extras/#extra-binds/)。这会将路径作为卷安装在你的 kubelet 中，可用于工作负载中的 hostPath 卷。 + +### 将 vSphere Cloud Provider 从树内迁移到树外 + +Kubernetes 正在逐渐不在树内维护云提供商。vSphere 有一个树外云提供商，可通过安装 vSphere 云提供商和云存储插件来使用。 + +有关如何从树内 vSphere 云提供商迁移到树外，以及如何在迁移后管理现有虚拟机，请参阅[此页面](../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 + +### 相关链接 + +- [Kubernetes 文档: 存储](https://kubernetes.io/docs/concepts/storage/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/install-cluster-autoscaler/install-cluster-autoscaler.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/install-cluster-autoscaler/install-cluster-autoscaler.md new file mode 100644 index 00000000000..9d03258c515 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/install-cluster-autoscaler/install-cluster-autoscaler.md @@ -0,0 +1,28 @@ +--- +title: Cluster Autoscaler +--- + + + + + +在本文中，你将学习如何使用 AWS EC2 Auto Scaling 组在 Rancher 自定义集群上安装和使用 [Kubernetes cluster-autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/)。 + +Cluster Autoscaler 是一个自动调整 Kubernetes 集群大小的工具。该工具在满足以下条件之一时能自动调整集群大小： + +* 集群中有 Pod 因资源不足而无法运行。 +* 集群中有一些节点长时间未得到充分利用，而且它们的 Pod 可以放到其他现有节点上。 + +为防止你的 pod 被驱逐，请在你的 pod 规范中设置 `priorityClassName: system-cluster-critical` 属性。 + +Cluster Autoscaler 运行在 Kubernetes master 节点上。它可以在 `kube-system` 命名空间中运行。Cluster Autoscaler 不会缩减运行非镜像 `kube-system` pod 的节点。 + +你可以在 worker 节点上运行 Cluster Autoscaler 的自定义 deployment，但需要小心以保证 Cluster Autoscaler 能正常运行。 + +## 云提供商 + +Cluster Autoscaler 为不同的云提供商提供支持。有关详细信息，请参见 [Cluster Autoscaler 支持的云提供商](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#deployment)。 + +### 在 Amazon 上设置 Cluster Autoscaler + +有关在 Amazon 上运行 Cluster Autoscaler 的详细信息，请参阅[此页面](use-aws-ec2-auto-scaling-groups.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/manage-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/manage-clusters.md new file mode 100644 index 00000000000..a1c89025444 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/manage-clusters.md @@ -0,0 +1,36 @@ +--- +title: 集群管理 +--- + + + + + +在 Rancher 中配置集群后，你可以开始使用强大的 Kubernetes 功能在开发、测试或生产环境中部署和扩展容器化应用。 + +:::note + +本节默认你已对 Docker 和 Kubernetes 有一定的了解。如果你需要了解 Kubernetes 组件如何协作，请参见 [Kubernetes 概念](../../../reference-guides/kubernetes-concepts.md)页面。 + +::: + +## 在 Rancher 中管理集群 + +将集群[配置到 Rancher](../kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md) 之后，[集群所有者](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#集群角色)需要管理这些集群。管理集群的选项如下： + +import ClusterCapabilitiesTable from '../../../shared-files/_cluster-capabilities-table.md'; + + + +## 配置工具 + +Rancher 包含 Kubernetes 中未包含的各种工具来协助你进行 DevOps 操作。Rancher 可以与外部服务集成，让你的集群更高效地运行。工具分为以下几类： + +- 告警 +- Notifiers +- Logging +- Monitoring +- Istio 服务网格 +- OPA Gatekeeper + +你可以通过 **Apps** 来安装工具。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/provisioning-storage-examples.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/provisioning-storage-examples.md new file mode 100644 index 00000000000..3e093ff09e4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/provisioning-storage-examples.md @@ -0,0 +1,15 @@ +--- +title: 配置存储示例 +--- + + + + + +Rancher 通过各种卷插件来支持持久存储。但是，在使用这些插件将持久存储绑定到工作负载之前，无论是使用云解决方案还是你自己管理的本地解决方案，你都必须先配置存储本身。 + +为了你的方便，Rancher 提供了配置主流存储的参考文档： + +- [NFS](nfs-storage.md) +- [vSphere](vsphere-storage.md) +- [EBS](persistent-storage-in-amazon-ebs.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/shared-files/_cluster-capabilities-table.md b/i18n/zh/docusaurus-plugin-content-docs/current/shared-files/_cluster-capabilities-table.md index c418e1eaf54..c4b68aa5d8c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/shared-files/_cluster-capabilities-table.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/shared-files/_cluster-capabilities-table.md @@ -2,13 +2,13 @@ | --- | --- | ---| ---|----| | [使用 kubectl 和 kubeconfig 文件来访问集群](../how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md) | ✓ | ✓ | ✓ | ✓ | | [管理集群成员](../how-to-guides/new-user-guides/manage-clusters/access-clusters/add-users-to-clusters.md) | ✓ | ✓ | ✓ | ✓ | -| [编辑和升级集群](../pages-for-subheaders/cluster-configuration.md) | ✓ | ✓ | ✓ | ✓2 | +| [编辑和升级集群](../reference-guides/cluster-configuration/cluster-configuration.md) | ✓ | ✓ | ✓ | ✓2 | | [管理节点](../how-to-guides/new-user-guides/manage-clusters/nodes-and-node-pools.md) | ✓ | ✓ | ✓ | ✓3 | -| [管理持久卷和存储类](../pages-for-subheaders/create-kubernetes-persistent-storage.md) | ✓ | ✓ | ✓ | ✓ | +| [管理持久卷和存储类](../how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md) | ✓ | ✓ | ✓ | ✓ | | [管理项目、命名空间和工作负载](../how-to-guides/new-user-guides/manage-clusters/projects-and-namespaces.md) | ✓ | ✓ | ✓ | ✓ | -| [使用应用目录](../pages-for-subheaders/helm-charts-in-rancher.md) | ✓ | ✓ | ✓ | ✓ | -| 配置工具（[Alerts、Notifiers、Monitoring](../pages-for-subheaders/monitoring-and-alerting.md)、[Logging](../pages-for-subheaders/logging.md) 和 [Istio](../pages-for-subheaders/istio.md)） | ✓ | ✓ | ✓ | ✓ | -| [运行安全扫描](../pages-for-subheaders/cis-scan-guides.md) | ✓ | ✓ | ✓ | ✓ | +| [使用应用目录](../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md) | ✓ | ✓ | ✓ | ✓ | +| 配置工具（[Alerts、Notifiers、Monitoring](../integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md)、[Logging](../integrations-in-rancher/logging/logging.md) 和 [Istio](../integrations-in-rancher/istio/istio.md)） | ✓ | ✓ | ✓ | ✓ | +| [运行安全扫描](../how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md) | ✓ | ✓ | ✓ | ✓ | | [轮换证书](../how-to-guides/new-user-guides/manage-clusters/rotate-certificates.md) | ✓ | ✓ | | | | [备份](../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher-launched-kubernetes-clusters.md)和[恢复](../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher-launched-kubernetes-clusters-from-backup.md) Rancher 启动的集群 | ✓ | ✓ | | ✓4 | | [在 Rancher 无法访问集群时清理 Kubernetes 组件](../how-to-guides/new-user-guides/manage-clusters/clean-cluster-nodes.md) | ✓ | | | | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8.json b/i18n/zh/docusaurus-plugin-content-docs/version-2.8.json index ebcffb97e2d..b4b95388418 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8.json +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8.json @@ -124,32 +124,40 @@ "description": "The label for category Infrastructure Setup in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Kubernetes Clusters in Rancher Setup": { - "message": "Rancher 设置中的 Kubernetes 集群", + "message": "Rancher 中的 Kubernetes 集群设置", "description": "The label for category Kubernetes Clusters in Rancher Setup in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Checklist for Production-Ready Clusters": { "message": "生产就绪集群检查清单", "description": "The label for category Checklist for Production-Ready Clusters in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Set Up Clusters from Hosted Kubernetes Providers": { + "sidebar.tutorialSidebar.category.Setting up Clusters from Hosted Kubernetes Providers": { "message": "通过托管 Kubernetes 提供商设置集群", - "description": "The label for category Set Up Clusters from Hosted Kubernetes Providers in sidebar tutorialSidebar" + "description": "The label for category Setting Up Clusters from Hosted Kubernetes Providers in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Launching Kubernetes on Windows Clusters": { + "message": "在 Windows 集群上启动 Kubernetes", + "description": "The label for category Launching Kubernetes on Windows Clusters in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Use Windows Clusters": { "message": "使用 Windows 集群", "description": "The label for category Use Windows Clusters in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Set Up Cloud Providers": { + "sidebar.tutorialSidebar.category.Setting up Cloud Providers": { "message": "设置 Cloud Provider", - "description": "The label for category Set Up Cloud Providers in sidebar tutorialSidebar" + "description": "The label for category Setting up Cloud Providers in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Launch Kubernetes with Rancher": { "message": "使用 Rancher 启动 Kubernetes", "description": "The label for category Launch Kubernetes with Rancher in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Use New Nodes in an Infra Provider": { + "sidebar.tutorialSidebar.category.Migrate to an Out-of-tree Cloud Provider": { + "message": "迁移到树外云提供商", + "description": "The label for category Migrate to an Out-of-tree Cloud Provider in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Launching New Nodes in an Infra Provider": { "message": "在基础设施提供商中使用新节点", - "description": "The label for category Use New Nodes in an Infra Provider in sidebar tutorialSidebar" + "description": "The label for category Launching New Nodes in an Infra Provider in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.vSphere": { "message": "vSphere", @@ -400,3 +408,4 @@ "description": "The label for category Container Security with Neuvector in sidebar tutorialSidebar" } } + \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md new file mode 100644 index 00000000000..de9bc25d31a --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md @@ -0,0 +1,130 @@ +--- +title: RKE 模板 +--- + + + + + +RKE 模板旨在让 DevOps 和安全团队标准化和简化 Kubernetes 集群创建的流程。 + +RKE 的全称是 [Rancher Kubernetes Engine](https://rancher.com/docs/rke/latest/en/)，它是 Rancher 用来配置 Kubernetes 集群的工具。 + +随着 Kubernetes 越来越受欢迎，管理更多小型集群逐渐成为趋势。如果你想要创建大量集群，对集群进行一致管理尤为重要。多集群管理面临着安全和附件配置执行的挑战，在将集群移交给最终用户之前，这些配置需要标准化。 + +RKE 模板有助于标准化这些配置。无论是使用 Rancher UI、Rancher API 还是自动化流程创建的集群，Rancher 都将保证从 RKE 集群模板创建的每个集群在生成方式上是一致的。 + +管理员可以控制最终用户能更改的集群选项。RKE 模板还可以与特定的用户和组共享，以便管理员可以为不同的用户集创建不同的 RKE 模板。 + +如果集群是使用 RKE 模板创建的，则不能让集群使用另一个 RKE 模板。你只能将集群更新为同一模板的新版本。 + +你可以[将现有集群的配置保存为 RKE 模板](apply-templates.md#将现有集群转换为使用-rke-模板)。这样，只有模板更新后才能更改集群的设置。新模板还可用于启动新集群。 + +RKE 模板的核心功能允许 DevOps 和安全团队： + +- 标准化集群配置并确保按照最佳实践创建 Rancher 配置的集群 +- 配置集群时，防止用户做出不明智的选择 +- 与不同的用户和组共享不同的模板 +- 将模板的所有权委托给受信任的用户进行更改 +- 控制哪些用户可以创建模板 +- 要求用户使用模板来创建集群 + +## 可配置的设置 + +RKE 模板可以在 Rancher UI 中创建或以 YAML 格式定义。当你使用 Rancher 从基础设施提供商配置自定义节点或一般节点时，它们可以指定为相同的参数： + +- 云提供商选项 +- Pod 安全选项 +- 网络提供商 +- Ingress Controller +- 网络安全配置 +- 网络插件 +- 私有镜像仓库 URL 和凭证 +- 附加组件 +- Kubernetes 选项，包括 kube-api、kube-controller、kubelet 和服务等 Kubernetes 组件的配置 + +RKE 模板的[附加组件](#附加组件)的功能特别强大，因为它允许多种自定义选项。 + +## RKE 模板的范围 + +Rancher 配置的集群支持 RKE 模板。模板可用于配置自定义集群或由基础设施提供商启动的集群。 + +RKE 模板用于定义 Kubernetes 和 Rancher 设置。节点模板负责配置节点。有关如何将 RKE 模板与硬件结合使用的参考，请参阅 [RKE 模板和硬件](infrastructure.md)。 + +可以从头开始创建 RKE 模板来预先定义集群配置。它们可以用于启动新集群，也可以从现有的 RKE 集群导出模板。 + +现有集群的设置可以[保存为 RKE 模板](apply-templates.md#将现有集群转换为使用-rke-模板)。这会创建一个新模板并将集群设置绑定到该模板。这样，集群只有在[模板更新](manage-rke1-templates.md#更新模板)的情况下才能[使用新版本的模板](manage-rke1-templates.md#升级集群以使用新的模板修订版)进行升级。新模板也可以用来创建新集群。 + + +## 示例场景 +如果一个组织同时拥有普通和高级 Rancher 用户，管理员可能希望为高级用户提供更多用于集群创建的选项，并限制普通用户的选项。 + +这些[示例场景](example-use-cases.md)描述组织如何使用模板来标准化集群创建。 + +示例场景包括： + +- **强制执行模板**：如果希望所有 Rancher 配置的新集群都具有某些设置，管理员可能想要[为每个用户强制执行一项或多项模板设置](example-use-cases.md#强制执行模板设置)。 +- **与不同的用户共享不同的模板**：管理员可以为[普通用户和高级用户提供不同的模板](example-use-cases.md#普通用户和高级用户模板)。这样，普通用户会有更多限制选项，而高级用户在创建集群时可以使用更多选项。 +- **更新模板设置**：如果组织的安全和 DevOps 团队决定将最佳实践嵌入到新集群所需的设置中，这些最佳实践可能会随着时间而改变。如果最佳实践发生变化，[可以将模板更新为新版本](example-use-cases.md#更新模板和集群)，这样，使用模板创建的集群可以[升级到模板的新版本](manage-rke1-templates.md#升级集群以使用新的模板修订版)。 +- **共享模板的所有权**：当模板所有者不再想要维护模板或想要共享模板的所有权时，此方案描述了如何[共享模板所有权](example-use-cases.md#允许其他用户控制和共享模板)。 + +## 模板管理 + +创建 RKE 模板时，可以在 Rancher UI 中的**集群管理**下的 **RKE 模板**中使用模板。创建模板后，你将成为模板所有者，这将授予你修改和共享模板的权限。你可以与特定用户或组共享 RKE 模板，也可以公开模板。 + +管理员可以开启模板强制执行，要求用户在创建集群时始终使用 RKE 模板。这使管理员可以保证 Rancher 总是创建指定配置的集群。 + +RKE 模板更新通过修订系统处理。如果要更改或更新模板，请创建模板的新版本。然后，可以将使用旧版本模板创建的集群升级到新模板修订版。 + +在 RKE 模板中，模板所有者可以限制设置的内容，也可以打开设置以供最终用户选择值。它们的差别体现在，创建模板时，Rancher UI 中的每个设置上的**允许用户覆盖**标示。 + +对于无法覆盖的设置，最终用户将无法直接编辑它们。为了让用户使用这些设置的不同选项，RKE 模板所有者需要创建 RKE 模板的新版本，这将允许用户升级和更改该选项。 + +本节中的文件解释了 RKE 模板管理的细节： + +- [获取创建模板的权限](creator-permissions.md) +- [创建和修改模板](manage-rke1-templates.md) +- [强制执行模板设置](enforce-templates.md#强制新集群使用-rke-模板) +- [覆盖模板设置](override-template-settings.md) +- [与集群创建者共享模板](access-or-share-templates.md#与特定用户或组共享模板) +- [共享模板的所有权](access-or-share-templates.md#共享模板所有权) + +你可以参见此[模板的示例 YAML 文件](../../../../reference-guides/rke1-template-example-yaml.md)作为参考。 + +## 应用模板 + +你可以使用你自己创建的模板来[创建集群](apply-templates.md#使用-rke-模板创建集群)，也可以使用[与你共享的模板](access-or-share-templates.md)来创建集群。 + +如果 RKE 模板所有者创建了模板的新版本，你可以[将你的集群升级到该版本](apply-templates.md#更新使用-rke-模板创建的集群)。 + +可以从头开始创建 RKE 模板来预先定义集群配置。它们可以用于启动新集群，也可以从现有的 RKE 集群导出模板。 + +你可以[将现有集群的配置保存为 RKE 模板](apply-templates.md#将现有集群转换为使用-rke-模板)。这样，只有模板更新后才能更改集群的设置。 + +## 标准化硬件 + +RKE 模板的目的是标准化 Kubernetes 和 Rancher 设置。如果你还想标准化你的基础设施，一个选择是将 RKE 模板与[其他工具](infrastructure.md)一起使用。 + +另一种选择是使用包含节点池配置选项，但不强制执行配置的[集群模板](../../manage-clusters/manage-cluster-templates.md)。 + +## YAML 定制 + +如果将 RKE 模板定义为 YAML 文件，则可以修改此[示例 RKE 模板 YAML](../../../../reference-guides/rke1-template-example-yaml.md)。RKE 模板中的 YAML 使用了 Rancher 在创建 RKE 集群时使用的相同自定义设置。但由于 YAML 要在 Rancher 配置的集群中使用，因此需要将 RKE 模板自定义项嵌套在 YAML 中的 `rancher_kubernetes_engine_config` 参数下。 + +RKE 文档也提供[注释的](https://rancher.com/docs/rke/latest/en/example-yamls/) `cluster.yml` 文件供你参考。 + +有关可用选项的更多信息，请参阅[集群配置](https://rancher.com/docs/rke/latest/en/config-options/)上的 RKE 文档。 + +### 附加组件 + +RKE 模板配置文件的附加组件部分的工作方式与[集群配置文件的附加组件部分](https://rancher.com/docs/rke/latest/en/config-options/add-ons/)相同。 + +用户定义的附加组件指令允许你调用和下拉 Kubernetes 清单或将它们直接内联。如果这些 YAML 清单包括在 RKE 模板中，Rancher 将在集群中部署这些 YAML 文件。 + +你可以使用附加组件执行以下操作： + +- 启动 Kubernetes 集群后，在集群上安装应用 +- 在使用 Kubernetes Daemonset 部署的节点上安装插件 +- 自动设置命名空间、ServiceAccount 或角色绑定 + +RKE 模板配置必须嵌套在 `rancher_kubernetes_engine_config` 参数中。要设置附加组件，在创建模板时单击**以 YAML 文件编辑**。然后使用 `addons` 指令添加清单，或使用 `addons_include` 指令设置哪些 YAML 文件可用于附加组件。有关自定义附加组件的更多信息，请参见[用户自定义附加组件文档](https://rancher.com/docs/rke/latest/en/config-options/add-ons/user-defined-add-ons/)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md new file mode 100644 index 00000000000..aa3c433b873 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md @@ -0,0 +1,11 @@ +--- +title: Kubernetes 集群基础设施 +--- + + + + + +要为具有外部数据库的高可用 K3s Kubernetes 集群设置基础设施，请参见[本页面](ha-k3s-kubernetes-cluster.md)。 + +要为高可用 RKE Kubernetes 集群设置基础设施，请参见[本页面](ha-rke1-kubernetes-cluster.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md new file mode 100644 index 00000000000..f8213ffbbac --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md @@ -0,0 +1,11 @@ +--- +title: "Kubernetes 使用教程" +--- + + + + + +本章节介绍如何安装 Kubernetes 集群，使得 Rancher Server 可以安装在该集群上。 + +Rancher 可以在任何 Kubernetes 集群上运行。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md new file mode 100644 index 00000000000..313585d60dd --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md @@ -0,0 +1,52 @@ +--- +title: 生产就绪集群检查清单 +--- + + + + + +本节将介绍创建生产就绪型 Kubernetes 集群的最佳实践。这个集群可用于运行你的应用和服务。 + +有关集群的要求（包括对 OS/Docker、硬件和网络的要求），请参阅[节点要求](../node-requirements-for-rancher-managed-clusters.md)部分。 + +本文介绍了我们推荐用于所有生产集群的最佳实践的简短列表。 + +如需获取推荐的所有最佳实践的完整列表，请参阅[最佳实践](../../../../reference-guides/best-practices/best-practices.md)部分。 + +### 节点要求 + +- 确保你的节点满足所有[节点要求](../node-requirements-for-rancher-managed-clusters.md)，包括端口要求。 + +### 备份 etcd + +* 启用 etcd 快照。验证是否正在创建快照，并执行灾难恢复方案，从而验证快照是否有效。etcd 是存储集群状态的位置，丢失 etcd 数据意味着丢失集群。因此，请确保为集群配置 etcd 的定期快照，并确保快照也是存储在外部（节点外）的。 + +### 集群架构 + +* 节点应具有以下角色配置之一： + * `etcd` + * `controlplane` + * `etcd` 和 `controlplane` + * `worker`（不应在具有 `etcd` 或 `controlplane` 角色的节点上使用或添加 `worker` 角色） +* 至少拥有三个角色为 `etcd` 的节点，来确保失去一个节点时仍能存活。增加 etcd 节点数量能提高容错率，而将 etcd 分散到不同可用区甚至能获取更好的容错能力。 +* 为两个或更多节点分配 `controlplane` 角色，能实现主组件的高可用性。 +* 为两个或多个节点分配 `worker` 角色，以便在节点故障时重新安排工作负载。 + +有关每个角色的用途的更多信息，请参阅 [Kubernetes 中的节点角色](roles-for-nodes-in-kubernetes.md)。 + +有关每个 Kubernetes 角色的节点数的详细信息，请参阅[推荐架构](../../../../reference-guides/rancher-manager-architecture/architecture-recommendations.md)部分。 + +### Logging 和 Monitoring + +* 为 Kubernetes 组件（系统服务）配置告警/通知程序。 +* 为集群分析和事后剖析配置 Logging。 + +### 可靠性 + +* 在集群上执行负载测试，以验证硬件是否可以支持你的工作负载。 + +### 网络 + +* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`，`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络（网络延迟特别高的情况下除外）中运行。 +* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区，这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 (../set-up-cloud-providers/set-up-cloud-providers.md) 资源，请查阅文档以了解限制（即区域存储限制）。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md new file mode 100644 index 00000000000..44c89d5068d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md @@ -0,0 +1,78 @@ +--- +title: Rancher 中的 Kubernetes 集群设置 +description: 配置 Kubernetes 集群 +--- + + + + + +Rancher 简化了集群的创建，允许你通过 Rancher UI 而不是更复杂的替代方案来创建集群。Rancher 为启动集群提供了多个选项。 请选择最适合您用例的选项。 + +本节假定你对 Docker 和 Kubernetes 有基本的了解。有关 Kubernetes 组件如何协同工作的简要说明，请参阅[概念](../../../reference-guides/kubernetes-concepts.md)页面。 + +有关 Rancher 服务如何配置集群以及使用哪些工具配置集群的概念性概述，请参阅[架构](../../../reference-guides/rancher-manager-architecture/rancher-manager-architecture.md)页面。 + +### 按集群类型划分的集群管理功能 + +下表总结了每种集群类型的可用选项和设置： + +import ClusterCapabilitiesTable from '../../../shared-files/\_cluster-capabilities-table.md'; + + + +## 在托管 Kubernetes 提供商中设置集群 + +在这种情况下，Rancher 不会配置 Kubernetes，因为它是由 Google Kubernetes Engine（GKE）、Amazon Elastic Container Service for Kubernetes 或 Azure Kubernetes Service 等提供商安装的。 + +如果使用 Google GKE 等 Kubernetes 提供商，Rancher 会与其云 API 集成，允许你从 Rancher UI 为托管集群创建和管理基于角色的访问控制。 + +更多信息，请参阅[托管 Kubernetes 集群](set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md)部分。 + +## 使用 Rancher 启动 Kubernetes + +在自己的节点上配置 Kubernetes 时，Rancher 使用 [Rancher Kubernetes Engine (RKE)](https://rancher.com/docs/rke/latest/en/) 作为库。RKE 是 Rancher 自己的轻量级 Kubernetes 安装程序。 + +在 RKE 集群中，Rancher 负责管理 Kubernetes 的部署。这些集群可以部署在任何裸机服务器、云提供商或虚拟化平台上。 + +这些节点可以通过 Rancher 的用户界面动态配置，它可以调用 [Docker Machine](https://docs.docker.com/machine/) 在各种云提供商上启动节点。 + +如果你已经有了一个要添加到 RKE 集群的节点，可以通过在该节点上运行 Rancher 代理容器将其添加到集群中。 + +更多信息，请参阅 [RKE 集群](../launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)部分。 + +### 在基础设施提供商中启动 Kubernetes 和配置节点 + +Rancher 可以在亚马逊 EC2、DigitalOcean、Azure 或 vSphere 等基础设施提供商中动态调配节点，然后在这些节点上安装 Kubernetes。 + +使用 Rancher，你可以基于[节点模板](../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md#node-templates)创建节点池。该模板定义了用于在云提供商中启动节点的参数。 + +使用基础设施提供商托管的节点的一个好处是，如果某个节点失去了与集群的连接，Rancher 可以自动替换掉它，从而保持预期的集群配置。 + +可用于创建节点模板的云提供商是由 Rancher UI 中激活的[节点驱动程序](../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md#node-drivers)决定的。 + +有关详细信息，请参阅[基础设施提供商托管的节点](../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)部分。 + +### 在已存在的自定义节点上启动 Kubernetes + +设置这种集群时，Rancher 会在已存在的[自定义节点](../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)上安装 Kubernetes，从而创建一个自定义集群。 + +你可以将任何节点添加到 Rancher 中，并用它们创建集群。 + +这些节点包括内部部署的裸机服务器、云托管的虚拟机或内部部署的虚拟机。 + +## 注册已存在的集群 + +集群注册功能取代了集群导入功能。 + +现在，注册 EKS 集群还能带来更多好处。在 Rancher UI 中，注册的 EKS 集群和在 Rancher 中创建的 EKS 集群在大多数情况下的处理方式相同，但删除除外。 + +删除在 Rancher 中创建的 EKS 集群时，集群将被销毁。删除在 Rancher 中注册的EKS集群时，该集群会断开与 Rancher 服务器的连接，但它仍然存在，你仍然可以按照在 Rancher 中注册前的方式访问它。 + +更多信息，请参阅[本页](register-existing-clusters.md)。 + +## 以编程方式创建集群 + +通过 Rancher 以编程方式部署 Kubernetes 集群的最常用方法是使用 Rancher2 Terraform 提供商。使用 Terraform 创建集群的文档在[这里](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster)。 + +可以使用 Terraform 创建或导入 EKS、GKE、AKS 集群和 RKE 集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-amazon.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-amazon.md new file mode 100644 index 00000000000..3f6a8b463d8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-amazon.md @@ -0,0 +1,194 @@ +--- +title: 将 Amazon 从树内迁移到树外 +--- + + + + + +Kubernetes 不再在树内维护云提供商。在 Kubernetes v1.27 及更高版本中，树内云提供商已被移除。当你从树内提供商迁移到树外提供商时，Rancher UI 允许你升级到 Kubernetes v1.27。 + +不过，如果你执行的是手动迁移，现有集群必须在迁移后升级到 Kubernetes v1.27 才能继续运行。 + +要从树内云提供商迁移到树外 AWS 云提供商，必须停止现有集群的 kube 控制器管理器，并安装 AWS 云控制器管理器。有许多方法可以做到这一点。有关详情，请参阅有关[外部云控制器管理器](https://cloud-provider-aws.sigs.k8s.io/getting_started/)的 AWS 官方文档。 + +如果可以接受迁移过程中出现一些停机，请按照说明[设置外部云提供商](../set-up-cloud-providers/amazon.md#using-the-out-of-tree-aws-cloud-provider)。这些说明概述了如何为新配置的集群配置树外云提供商。在设置过程中，会有一些停机，因为从旧云提供商停止运行到新云提供商开始运行之间会有一段时间的间隔。 + +如果您的设置不能容忍任何控制平面停机，则必须启用领导者迁移。这有助于从 kube 控制器管理器中的控制器顺利过渡到云控制器管理器中的对应控制器。有关详细信息，请参阅 AWS 官方文档[使用领导者迁移](https://cloud-provider-aws.sigs.k8s.io/getting_started/)。 + +:::note Important: +Kubernetes [云控制器迁移文档](https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/#before-you-begin)指出，可以使用相同的 Kubernetes 版本进行迁移，但假设迁移是 Kubernetes 升级的一部分。请参考有关[迁移到要使用的云控制器管理器](https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/)的 Kubernetes 文档，了解迁移前是否需要自定义设置。确认[迁移配置值](https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/#default-configuration)。如果您的云提供商提供 Node IPAM 控制器的实现，您还需要迁移 [IPAM 控制器](https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/#node-ipam-controller-migration)。 +::: + + + + +1. 更新集群配置，启用领导者迁移： + +```yaml +spec: + rkeConfig: + machineSelectorConfig: + - config: + kube-controller-manager-arg: + - enable-leader-migration + machineLabelSelector: + matchExpressions: + - key: rke.cattle.io/control-plane-role + operator: In + values: + - "true" +``` + +请注意，云提供商在此步骤中仍是`aws` + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + cloud-provider-name: aws +``` + +2. 标记控制平面节点不可调度，以便 AWS 云控制器 pod 仅在升级到外部云提供商后才在节点上运行： + +```shell +kubectl cordon -l "node-role.kubernetes.io/control-plane=true" +``` + +3. 要安装启用了领导者迁移的 AWS 云控制器管理器，请遵循[部署云控制器管理器 Chart](../set-up-cloud-providers/amazon.md#using-the-out-of-tree-aws-cloud-provider) 的步骤 1-3。从 Kubernetes 1.22 起，kube-controller-manager 将使用默认配置，该配置将满足 controller-to-manager 的迁移。更新 `spec.rkeConfig.additionalManifest` 下 `aws-cloud-controller-manager` 的容器参数，以启用领导者迁移： + +```shell +- '--enable-leader-migration=true' +``` + +4. 安装 Chart 并确认 Daemonset `aws-cloud-controller-manager` 已成功部署： + +```shell +kubectl rollout status daemonset -n kube-system aws-cloud-controller-manager +``` + +5. 更新配置集群以更改云提供商，并从 kube 控制器中移除领导者迁移 参数。如果升级 Kubernetes 版本，也要在集群 YAML 文件的 `spec.kubernetesVersion` 部分设置 Kubernetes 版本 + +:::note Important + +如果不依赖 rke2 监管程序正确设置提供商 ID，则只移除 `cloud-provider-name: aws`。 + +::: + +如果不想在集群中启用 `enable-leader-migration` 功能，请将其移除： + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + cloud-provider-name: external +``` + +移除 `enable-leader-migration`： + +```yaml +spec: + rkeConfig: + machineSelectorConfig: + - config: + kube-controller-manager-arg: + - enable-leader-migration + machineLabelSelector: + matchExpressions: + - key: rke.cattle.io/control-plane-role + operator: In + values: + - "true" +``` + +:::tip + +您也可以在升级后禁用领导者迁移，因为由于只有一个云控制器管理器，不再需要领导者迁移，可以将其移除。升级 Chart 并从容器参数中删除以下部分： + +```yaml +- --enable-leader-migration=true +``` + +::: + +使用以下命令验证云控制器管理器更新是否成功上线： + +```shell +kubectl rollout status daemonset -n kube-system aws-cloud-controller-manager +``` + +6. 云提供商负责设置节点的 ProviderID。检查是否所有节点都用 ProviderID 进行了初始化： + +```shell +kubectl describe nodes | grep "ProviderID" +``` + + + + + +1. 更新集群配置，在 `cluster.yml` 中启用领导者迁移： + +```yaml +services: + kube-controller: + extra_args: + enable-leader-migration: "true" +``` + +请注意，云提供商在此步骤中仍是`aws` + +```yaml +cloud_provider: + name: aws +``` + +2. 标记控制平面节点不可调度，以便 AWS 云控制器 pod 仅在升级到外部云提供商后才在节点上运行： + +```shell +kubectl cordon -l "node-role.kubernetes.io/controlplane=true" +``` + +3. 要安装 AWS 云控制器管理器，必须启用领导者迁移，并遵循在新集群上安装 AWS 时的相同步骤。要启用领导者迁移，请在步骤 7 中的容器参数中添加以下内容，同时[按照步骤安装 Chart](../set-up-cloud-providers/amazon.md#helm-chart-installation-from-ui)： + +```yaml +- "--enable-leader-migration=true" +``` + +4. 确认 Chart 已安装，但由于控制面板节点不可调度，新的 pod 尚未运行。在下一步更新集群后，RKE 将升级并允许对每个节点的调度，并调度 `aws-controller-manager` pod。 + +5. 更新 `cluster.yml`以更改云提供商，并从 kube-controller 中移除领导者迁移参数。 + +选择 **External Amazon (out-of-tree)** 可设置 `--cloud-provider=external`，并启用 `useInstanceMetadataHostname`。节点驱动集群和自定义集群必须启用 `useInstanceMetadataHostname`，否则无法通过 `--node-name`提供自定义节点名称。启用 `useInstanceMetadataHostname` 会查询 ec2 元数据服务，并为 `kubelet` 和 `kube-proxy`将 `/hostname` 设置为 `hostname-override` ： + +```yaml +rancher_kubernetes_engine_config: + cloud_provider: + name: external-aws + useInstanceMetadataHostname: true/false +``` + +如果不想在集群中启用 `enable-leader-migration`，请移除它： + +```yaml +services: + kube-controller: + extra_args: + enable-leader-migration: "true" +``` + +:::tip +完成迁移后，还可以禁用领导者迁移。升级 Chart 并从容器参数中删除以下部分： + +```yaml +- --enable-leader-migration=true +``` + +::: + +6. 如果要升级集群的 Kubernetes 版本，也要设置 Kubernetes 版本。 + +7. 更新集群。现在，`aws-cloud-controller-manager` pod 应该已经运行。 + + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md new file mode 100644 index 00000000000..3649e928567 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md @@ -0,0 +1,111 @@ +--- +title: 将 vSphere 从树内迁移到树外 +--- + + + + + +Kubernetes 不再在树内维护云提供商。vSphere 有一个树外云提供商，可以通过安装 vSphere 云提供商和云存储插件来使用。 + +本页面介绍如何从树内 vSphere 云提供商迁移到树外，并在迁移后管理现有虚拟机。 + +它遵循官方 [vSphere 迁移文档](https://vsphere-csi-driver.sigs.k8s.io/features/vsphere_csi_migration.html)中提供的步骤，并提供在 Rancher 中执行的步骤。 + +### Cloud-config 格式限制 + +由于 vSphere Cloud Storage Interface (CSI) 中的一个现有错误，使用以下 cloud-config 格式配置的现有卷将无法迁移。 + +如果 cloud-config 中的数据存储和资源池路径采用这种格式，vsphere CSI 驱动程序将无法识别它： + +```yaml +default-datastore: /datastore/ +resourcepool-path: "/host//Resources/" +``` + +使用树内提供商以下列格式配置的卷将正确迁移： + +```yaml +default-datastore: +resourcepool-path: "/Resources/" +``` + +上游 bug: https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/628 + +Rancher 问题跟踪此 bug: https://github.com/rancher/rancher/issues/31105 + +## 前提条件 + +- vSphere CSI 迁移需要 vSphere 7.0u1。为了能够管理现有树内 vSphere 卷，请将 vSphere 升级到 7.0u1。 +- Kubernetes 版本必须为 1.19 或更高。 + +## 迁移 + +### 1. 安装 CPI 插件 + +安装 CPI 之前，我们需要用 `node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule` 标记所有节点。 + +可以通过运行以下命令来完成： + +``` +curl -O https://raw.githubusercontent.com/rancher/helm3-charts/56b622f519728378abeddfe95074f1b87ab73b1e/charts/vsphere-cpi/taints.sh +``` + +或: + +``` +wget https://raw.githubusercontent.com/rancher/helm3-charts/56b622f519728378abeddfe95074f1b87ab73b1e/charts/vsphere-cpi/taints.sh +chmod +x taints.sh +./taints.sh +``` + +运行脚本并标记所有节点后，启动 Helm vSphere CPI Chart。 + +1. 点击 **☰ > Cluster Management**. +1. 转到将要安装 vSphere CPI Chart 的集群，然后点击 **Explore**. +1. 点击 **Apps > Charts**. +1. 点击 **vSphere CPI**. +1. 点击 **Install**. +1. 填写所需的 vCenter 详细信息，然后单击 **Install**. + +vSphere CPI 会使用 ProviderID 初始化所有节点，这是 vSphere CSI 驱动程序所需要的。 + +使用以下命令检查是否用 ProviderID 初始化了所有节点： + +``` +kubectl describe nodes | grep "ProviderID" +``` + +### 2. 安装 CSI 驱动程序 + +1. 点击 **☰ > Cluster Management**. +1. 转到将要安装 vSphere CSI Chart 的集群，点击 **Explore**. +1. 点击 **Apps > Charts**. +1. 点击 **vSphere CSI**. +1. 点击 **Install**. +1. 填写所需的 vCenter 详细信息，然后单击 **Install**. +1. 选中 **Customize Helm options before install**，然后单击 **Next**. +1. 在 **Features** 选项卡上， 选中 **Enable CSI Migration**. +1. （可选）转到 **Storage** 选项卡设置数据存储。此 Chart 会创建一个以 `csi.vsphere.vmware.com` 作为供应商的 StorageClass。在创建此 StorageClass 时，您可以提供用于 CSI 卷配置的数据存储的 URL。数据存储 URL 可在 vSphere 客户端中通过选择数据存储并转到摘要选项卡找到。填写 StorageClass 的详细信息。 +1. 点击 **Install**. + +### 3. 编辑集群以启用 CSI 迁移功能标志 + +1. 编辑集群时，如果 Kubernetes 版本低于 1.19，请从 **Kubernetes Version** 下拉菜单中选择 1.19 或更高版本。 +2. 要启用功能标志，请单击 "Edit as YAML", 然后在 kube-controller 和 kubelet 下添加以下内容: + + ```yaml + extra_args: + feature-gates: "CSIMigration=true,CSIMigrationvSphere=true" + ``` + +### 4. 清空工作节点 + +在升级过程中，必须先清空工作节点，然后再更改kubelet和kube-controller-manager参数。 + +1. 点击 **☰ > Cluster Management**。 +1. 转到要清空工作节点的集群，点击 **⋮ > Edit Config**。 +1. 在 **Advanced Options** 部分，将 **Maximum Worker Nodes Unavailable** 字段设为 1。 +1. 要在升级期间清空节点，请选择 **Drain Nodes > Yes**. +1. 将 **Force** 和 **Delete Local Data** 设置为 **true**. +1. 点击 **Save** 升级集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md new file mode 100644 index 00000000000..f6aa2871a8c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md @@ -0,0 +1,47 @@ +--- +title: 设置云提供商 +--- + + + + + +_cloud provider_ 是 Kubernetes 中的一个模块，它提供了一个用于管理节点、负载均衡器和网络路由的接口。 + +在 Rancher 中设置 cloud provider 时，如果你使用的云提供商支持自动化，Rancher Server 可以在启动 Kubernetes 定义时自动配置新节点、负载均衡器或持久存储设备。 + +如果你配置的节点云提供商集群不满足先决条件，集群将无法正确配置。 + +**Cloud Provider** 选项默认设置为 `None`。 + +可以启用的云提供商包括： + +* Amazon +* Azure +* GCE (Google Compute Engine) +* vSphere + +### 设置 Amazon 云提供商 + +有关启用 Amazon 云提供商的详细信息，请参阅[此页面](amazon.md)。 + +### 设置 Azure 云提供商 + +有关启用 Azure 云提供商的详细信息，请参阅[此页面](azure.md)。 + +### 设置 GCE 云提供商 + +有关启用 Google Compute Engine 云提供商的详细信息，请参阅[此页面](google-compute-engine.md)。 + +### 设置 vSphere 云提供商 + +有关启用 vSphere 云提供商的详细信息，请参阅[树内 vSphere 配置](configure-in-tree-vsphere.md) 和 [树外 vSphere 配置](configure-out-of-tree-vsphere.md)。 + +### 设置自定义云提供商 + +任何 Kubernetes Cloud Provider 都可以通过`自定义`云提供商进行配置。 + +对于自定义云提供商选项，你可以参考 [RKE 文档](https://rancher.com/docs/rke/latest/en/config-options/cloud-providers/)，了解如何为你的云提供商编辑 yaml 文件。特定云提供商的详细配置说明如下： + +- [vSphere](https://rke.docs.rancher.com/config-options/cloud-providers/vsphere) +- [OpenStack](https://rancher.com/docs/rke/latest/en/config-options/cloud-providers/openstack/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md new file mode 100644 index 00000000000..09d36c82e7c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md @@ -0,0 +1,284 @@ +--- +title: 在 Windows 集群上启动 Kubernetes +--- + + + + + +使用 Rancher 配置[自定义集群](../../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)时，Rancher 通过 RKE（Rancher Kubernetes Engine）在现有节点上安装 Kubernetes。 + +在使用 Rancher 配置的 Windows 集群中，集群必须同时包含 Linux 和 Windows 节点。Kubernetes controlplane 只能运行在 Linux 节点上，Windows 节点只能有 Worker 角色。Windows 节点只能用于部署工作负载。 + +Windows 集群的其他要求如下： + +- 只有在创建集群时启用了 Windows 支持的集群才能添加 Windows 节点。无法为现有集群启用 Windows 支持。 +- 需要 Kubernetes 1.15+。 +- 必须使用 Flannel 网络提供商。 +- Windows 节点必须有 50 GB 的磁盘空间。 + +有关完整的要求列表，请参阅[本节](#windows-集群的要求)。 + +有关支持 Windows 的 Kubernetes 功能摘要，请参阅[在 Windows 中使用 Kubernetes 支持的功能和限制](https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#supported-functionality-and-limitations) 的 Kubernetes 文档，或[在 Kubernetes 中调度 Windows 容器的指南](https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-containers/)。 + +### RKE2 Windows + +RKE2 配置功能还包括在 Windows 集群上安装 RKE2。RKE2 的 Windows 功能包括： + +- 由 containerd 提供支持的使用 RKE2 的 Windows 容器 +- 直接从 Rancher UI 配置 Windows RKE2 自定义集群 +- 用于 Windows RKE2 自定义集群的 Calico CNI +- 技术预览包含了 Windows Server 的 SAC 版本（2004 和 20H2） + +要使 Windows 支持 RKE2 自定义集群，请选择 Calico 作为 CNI。 + +:::note + +默认情况下，Rancher 允许 Windows 工作负载 pod 部署在 Windows 和 Linux Worker 节点上。在 RKE2 中创建混合集群时，你必须编辑 Chart 中的 `nodeSelector`，从而将 Pod 放置到兼容的 Windows 节点上。有关如何使用 `nodeSelector` 将 pod 分配给节点的更多信息，请参阅 [Kubernetes 文档](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)。 + +::: + +- Kubernetes v1.24.1 及更高版本支持 Windows RKE2 中的 HostProcess 容器。有关详细信息，请参阅[上游文档](https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/)。 + +## Windows 集群的要求 + +网络、操作系统和 Docker 的一般节点要求与 [Rancher 安装](../../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md) 的节点要求相同 + +### 操作系统和 Docker 要求 + +我们对 Windows Server 和 Windows 容器的支持与 LTSC（长期服务渠道）和 SAC（半年渠道）的 Microsoft 官方生命周期相匹配。 + +有关 Windows Server 的支持生命周期的日期，请参阅 [Microsoft 文档](https://docs.microsoft.com/en-us/windows-server/get-started/windows-server-release-info)。 + +### Kubernetes 版本 + +需要 Kubernetes v1.15+。 + +如果你在 Windows Server 20H2 Standard Core 上使用 Kubernetes v1.21，则必须在节点上安装补丁“2019-08 Servicing Stack Update for Windows Server”。 + +### 节点要求 + +集群中的主机至少需要： + +- 2 核 CPU +- 5 GB 内存 +- 50 GB 磁盘空间 + +Rancher 不会配置不满足要求的节点。 + +### 网络要求 + +在配置新集群之前，请确保你已经在接收入站网络流量的设备上安装了 Rancher。这是集群节点与 Rancher 通信所必需的。如果你尚未安装 Rancher，请在继续阅读本指南之前先参阅[安装文档](../../../../getting-started/installation-and-upgrade/installation-and-upgrade.md)进行安装。 + +Rancher 仅支持使用 Flannel 作为网络提供商的 Windows。 + +有两个网络选项：[**Host Gateway (L2bridge)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) 和 [**VXLAN (Overlay)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan)。 默认选项是 **VXLAN (Overlay)** 模式。 + +对于 **Host Gateway (L2bridge)** 网络，最好为所有节点使用相同的第 2 层网络。否则，你需要为它们配置路由规则。有关详细信息，请参阅[配置云托管 VM 路由的文档](network-requirements-for-host-gateway.md#云托管虚拟机的路由配置)。如果你使用的是 Amazon EC2、Google GCE 或 Azure 虚拟机，你需要[禁用私有 IP 地址检查](network-requirements-for-host-gateway.md#禁用私有-ip-地址检查)。 + +对于 **VXLAN (Overlay)** 网络, 必须安装 [KB4489899](https://support.microsoft.com/en-us/help/4489899) 补丁。大多数云托管虚拟机已经安装了此修复程序。 + +如果你在为 AWS 虚拟私有云配置 DHCP 选项集，请注意，你只能在 `domain-name` 选项字段中指定一个域名。详情请参见 [DHCP 选项文档](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html)。 + +:::note + +一些 Linux 操作系统支持以空格分隔的多个域名。但是，其他 Linux 操作系统和 Windows 将该值视为单个域名，从而导致意外错误。如果你的 DHCP 选项集与具有多个操作系统实例的 VPC 相关联，请仅指定一个域名。 + +::: + +### 带有 ESXi 6.7u2 及更高版本的 vSphere 上的 Rancher + +如果你在带有 ESXi 6.7u2 或更高版本的 VMware vSphere 上使用 Rancher，并使用 Red Hat Enterprise Linux 8.3、CentOS 8.3 或 SUSE Enterprise Linux 15 SP2 或更高版本，你需要禁用 `vmxnet3` 虚拟网络适配器硬件卸载功能。否则，不同集群节点上的 pod 之间的所有网络连接会因为超时错误而失败。从 Windows pod 到在 Linux 节点上运行的关键服务（例如 CoreDNS）的所有连接也将失败。外部连接也可能失败。出现这个问题的原因是 Linux 发行版在 `vmxnet3` 中启用了硬件卸载功能，而且 `vmxnet3` 硬件卸载功能中存在一个会丢弃客户覆盖流量的数据包的 bug。要解决此问题，必须禁用 `vmxnet3` 硬件卸载功能。此设置不会在重启后继续生效，因此需要在每次启动时禁用。推荐的做法是在 `/etc/systemd/system/disable_hw_offloading.service` 中创建一个 systemd 单元文件，这会在启动时禁用 `vmxnet3` 硬件卸载功能。禁用 `vmxnet3` 硬件卸载功能的示例 systemd 单元文件如下所示。注意，`` 必须自定义为主机的 `vmxnet3` 网络接口，如 `ens192`： + +``` +[Unit] +Description=Disable vmxnet3 hardware offloading feature + +[Service] +Type=oneshot +ExecStart=ethtool -K tx-udp_tnl-segmentation off +ExecStart=ethtool -K tx-udp_tnl-csum-segmentation off +StandardOutput=journal + +[Install] +WantedBy=multi-user.target +``` + +然后在 systemd 单元文件上设置适当的权限： + +``` +chmod 0644 /etc/systemd/system/disable_hw_offloading.service +``` + +最后，启用 systemd 服务： + +``` +systemctl enable disable_hw_offloading.service +``` + +### 架构要求 + +Kubernetes 集群管理节点（`etcd` 和 `controlplane`）必须运行在 Linux 节点上。 + +部署工作负载的 `worker` 节点通常是 Windows 节点，但必须至少有一个 `worker` 节点运行在 Linux 上，才能按顺序运行 Rancher Cluster Agent、DNS、Metrics Server 和 Ingress 相关容器。 + +#### 推荐架构 + +我们推荐下表中列出的三节点架构，但你始终可以添加额外的 Linux 和 Windows worker 节点来扩展集群，从而实现冗余： + +| 节点 | 操作系统 | Kubernetes 集群角色 | 用途 | +| ------ | --------------------------------------------- | --------------------------- | ----------------------------------------------------------- | +| 节点 1 | Linux (推荐 Ubuntu Server 18.04) | Control plane, etcd, worker | 管理 Kubernetes 集群 | +| 节点 2 | Linux (推荐 Ubuntu Server 18.04) | Worker | 支持集群的 Rancher 集群代理、Metrics 服务器、DNS 和 Ingress | +| 节点 3 | Windows（Windows Server 核心版本 1809 或更高版本） | Worker | 运行 Windows 容器 | + +### 容器要求 + +Windows 要求容器的版本必须与部署容器的 Windows Server 的版本一致。因此，你必须在 Windows Server 核心版本 1809 或更高版本上构建容器。如果你已经使用早期的 Windows Server 核心版本构建了容器，则必须使用 Windows Server 核心版本 1809 或更高版本重新构建容器。 + +### 云提供商特定要求 + +如果你在集群中设置了 Kubernetes 云提供商，则需要进行一些额外的操作。如果你想使用云提供商的功能，例如为集群自动配置存储、负载均衡器或其他基础设施，你可能需要设置云提供商。有关如何配置满足条件的云提供商集群节点，请参阅[此页面](../set-up-cloud-providers/set-up-cloud-providers.md)。 + +如果你的云提供商是 GCE（Google Compute Engine），则必须执行以下操作： + +- 按照[步骤](../set-up-cloud-providers/google-compute-engine.md)在`cluster.yml` 中启用 GCE 云提供商。 +- 在 Rancher 中配置集群时，在 Rancher UI 中选择**自定义云提供商**作为云提供商。 + +## 教程： 如何创建支持 Windows 的集群 + +本教程描述了如何使用[推荐架构](#推荐架构)中的三个节点创建由 Rancher 配置的集群。 + +在在现有节点上使用 Rancher 配置集群时，你需要在每个节点上安装 [Rancher Agent](../../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/rancher-agent-options.md)来将节点添加到集群中。在 Rancher UI 中创建或编辑集群时，你会看到一个**自定义节点运行命令**，你可以在每台服务器上运行该命令，从而将服务器添加到集群中。 + +要设置支持 Windows 节点和容器的集群，你需要完成以下任务： + +### 1. 配置主机 + +要在具有 Windows 支持的现有节点上配置集群，请准备好你的主机。 + +主机可以是： + +- 云托管的虚拟机 +- 虚拟化集群中的虚拟机 +- 裸金属服务器 + +你将配置三个节点： + +- 一个 Linux 节点，用于管理 Kubernetes controlplane 并存储你的 `etcd`。 +- 第二个 Linux 节点，它将作为 worker 节点。 +- Windows 节点，它将作为 worker 节点运行 Windows 容器。 + +| 节点 | 操作系统 | +| ------ | --------------------------------------------- | +| 节点 1 | Linux (推荐 Ubuntu Server 18.04) | +| 节点 2 | Linux (推荐 Ubuntu Server 18.04) | +| 节点 3 | Windows（Windows Server 核心版本 1809 或更高版本） | + +如果你的节点托管在**云提供商**上，并且你需要自动化支持（例如负载均衡器或持久存储设备），你的节点还需要满足额外的配置要求。详情请参见[选择云提供商](../set-up-cloud-providers/set-up-cloud-providers.md)。 + +### 2. 在现有节点上创建集群 + +在现有节点上创建 Windows 集群的说明与一般[创建自定义集群的说明](../../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)非常相似，但有一些特定于 Windows 的要求。 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面上，单击**创建**。 +1. 单击**自定义**。 +1. 在**集群名称**字段中输入集群的名称。 +1. 在 **Kubernetes 版本**下拉菜单中，选择 v1.19 或更高版本。 +1. 在**网络提供商**字段中，选择 **Flannel**。 +1. 在 **Windows 支持**中，单击**启用**。 +1. 可选：启用 Windows 支持后，你将能够选择 Flannel 后端模式。有两个网络选项：[**Host Gateway (L2bridge)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) 和 [**VXLAN (Overlay)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan)。 默认选项是 **VXLAN (Overlay)** 模式。 +1. 点击**下一步**。 + +:::note 重要提示： + +对于 Host Gateway (L2bridge) 网络，最好为所有节点使用相同的第 2 层网络。否则，你需要为它们配置路由规则。有关详细信息，请参阅[配置云托管 VM 路由的文档](network-requirements-for-host-gateway.md#云托管虚拟机的路由配置)。如果你使用的是 Amazon EC2、Google GCE 或 Azure 虚拟机，你需要[禁用私有 IP 地址检查](network-requirements-for-host-gateway.md#禁用私有-ip-地址检查)。 + +::: + +### 3. 将节点添加到集群 + +本节介绍如何将 Linux 和 Worker 节点注册到集群。你将在每个节点上运行一个命令，该命令将安装 Rancher Agent 并允许 Rancher 管理每个节点。 + +#### 添加 Linux master 节点 + +在本节中，你需要在 Rancher UI 上填写表单以获取自定义命令，从而在 Linux master 节点上安装 Rancher Agent。然后，复制该命令并在 Linux master 节点上运行命令，从而在集群中注册该节点。 + +集群中的第一个节点应该是具有 **controlplane** 和 **etcd** 角色的 Linux 主机。至少必须为此节点启用这两个角色，并且必须先将此节点添加到集群中，然后才能添加 Windows 主机。 + +1. 在**节点操作系统**中，单击 **Linux**。 +1. 在**节点角色**中，至少选择 **etcd** 和 **controlplane**。推荐选择所有的三个角色。 +1. 可选：如果点击**显示高级选项**，你可以自定义 [Rancher Agent](../../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/rancher-agent-options.md)和[节点标签](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)的设置。 +1. 将屏幕上显示的命令复制到剪贴板。 +1. SSH 到你的 Linux 主机，然后运行复制到剪贴板的命令。 +1. 完成配置 Linux 节点后，选择**完成**。 + +**结果**： + +你已创建集群，集群的状态是**配置中**。Rancher 已在你的集群中。 + +当集群状态变为 **Active** 后，你可访问集群。 + +**Active** 状态的集群会分配到两个项目： + +- `Default`：包含 `default` 命名空间 +- `System`：包含 `cattle-system`，`ingress-nginx`，`kube-public` 和 `kube-system` 命名空间。 + + +节点可能需要几分钟才能注册到集群中。 + +#### 添加 Linux Worker 节点 + +在本节中，我们通过运行命令将 Linux Worker 节点注册到集群中。 + +在初始配置集群之后，你的集群只有一个 Linux 主机。接下来，我们添加另一个 Linux `worker` 主机，用于支持集群的 _Rancher Cluster Agent_、_Metrics Server_、_DNS_ 和 _Ingress_。 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 转到你创建的集群，然后单击 **⋮ > 编辑配置**。 +1. 向下滚动到**节点操作系统**。选择 **Linux**。 +1. 在**自定义节点运行命令**中，转到**节点选项**并选择 **Worker** 角色。 +1. 将屏幕上显示的命令复制到剪贴板。 +1. 使用远程终端连接登录到你的 Linux 主机。粘贴剪贴板的命令并运行。 +1. 在 **Rancher**中，单击**保存**。 + +**结果**：**Worker** 角色已安装在你的 Linux 主机上，并且节点会向 Rancher 注册。节点可能需要几分钟才能注册到集群中。 + +:::note + +Linux Worker 节点上的污点 + +以下污点将添加集群中的 Linux Worker 节点中。将此污点添加到 Linux Worker 节点后，添加到 Windows 集群的任何工作负载都将自动调度到 Windows Worker 节点。如果想将工作负载专门调度到 Linux Worker 节点上，则需要为这些工作负载添加容忍度。 + +| 污点键 | 污点值 | 污点效果 | +| -------------- | ---------- | ------------ | +| `cattle.io/os` | `linux` | `NoSchedule` | + +::: + +#### 添加 Windows Worker 节点 + +在本节中，我们通过运行命令将 Windows Worker 节点注册到集群中。 + +你可以通过编辑集群并选择 **Windows** 选项，从而将 Windows 主机添加到集群中。 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 转到你创建的集群，然后单击 **⋮ > 编辑配置**。 +1. 向下滚动到**节点操作系统**。选择 **Windows**。注意：你将看到 **worker** 角色是唯一可用的角色。 +1. 将屏幕上显示的命令复制到剪贴板。 +1. 使用你喜欢的工具（例如 [Microsoft 远程桌面](https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/remote-desktop-clients)）登登录到 Windows 主机。在 **Command Prompt (CMD)** 中运行复制到剪贴板的命令。 +1. 在 Rancher 中，单击**保存**。 +1. 可选：如果要向集群添加更多 Windows 节点，请重复这些操作。 + +**结果**：**Worker** 角色已安装在你的 Windows 主机上，并且节点会向 Rancher 注册。节点可能需要几分钟才能注册到集群中。你现在已拥有一个 Windows Kubernetes 集群。 + +### 可选的后续步骤 + +创建集群后，你可以通过 Rancher UI 访问集群。最佳实践建议你设置以下访问集群的备用方式： + +- **通过 kubectl CLI 访问你的集群**：按照[这些步骤](../../manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#在工作站使用-kubectl-访问集群)在你的工作站上使用 kubectl 访问集群。在这种情况下，你将通过 Rancher Server 的身份验证代理进行身份验证，然后 Rancher 会让你连接到下游集群。此方法允许你在没有 Rancher UI 的情况下管理集群。 +- **通过 kubectl CLI 使用授权的集群端点访问你的集群**：按照[这些步骤](../../manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#直接使用下游集群进行身份验证)直接使用 kubectl 访问集群，而无需通过 Rancher Server 进行身份验证。我们建议设置此替代方法来访问集群，以便在无法连接到 Rancher 时访问集群。 + +## Azure 中存储类的配置 + +如果你的节点使用 Azure VM，则可以使用 [Azure 文件](https://docs.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv)作为集群的存储类（StorageClass）。详情请参见[此部分](azure-storageclass-configuration.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md new file mode 100644 index 00000000000..7710a20681c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md @@ -0,0 +1,65 @@ +--- +title: 集群访问 +--- + + + + + +本节介绍可以用来访问 Rancher 管理的集群的工具。 + +有关如何授予用户访问集群的权限的信息，请参阅[将用户添加到集群](add-users-to-clusters.md)部分。 + +有关 RBAC 的更多信息，请参阅[本节](../../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md)。 + +有关如何设置认证的信息，请参阅[本节](../../authentication-permissions-and-global-configuration/authentication-config/authentication-config.md)。 + +## Rancher UI 中的集群 + +我们提供了多种通过 Rancher UI 查看和管理集群的方法。 + +### 集群页面 + +从 **☰** 菜单访问**集群**页面： + +1. 单击 **☰**。 +1. 选择**集群管理**。 + +你还可以通过单击 Rancher UI **主页**集群表格上方的**管理**按钮来访问**集群**页面。 + +在**集群**页面上，选择每行末尾的 **⁝** 以查看包含以下选项的子菜单： + +* [Kubectl Shell](use-kubectl-and-kubeconfig.md) +* 下载 KubeConfig +* 将 KubeConfig 复制到剪切板 +* 编辑配置 +* 查看 YAML +* 下载 YAML + +## 集群仪表板 + +在**集群**页面上，选择每行末尾的**浏览**按钮查看该集群的**集群仪表板**。你还可以通过单击表中集群的名称，然后单击**集群**页面上的**浏览**按钮来查看仪表板。 + +也可以通过单击集群名称从 Rancher UI **主页**访问**集群仪表板**。 + +你还可以从顶部导航栏中的 **☰** 访问**集群仪表板**： + +1. 单击 **☰**。 +1. 从**浏览集群**菜单中选择集群的名称。 + +**集群仪表板**列出了集群相关的信息，例如节点数量、内存使用情况、事件和资源。 + +## kubectl + +你可以使用 Kubernetes 命令行工具 [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)，来管理你的集群。使用 kubectl 有两种选择： + +- **Rancher kubectl shell**：通过启动 Rancher UI 中可用的 kubectl shell 与集群交互。此选项不需要你进行任何配置操作。有关详细信息，请参阅[使用 kubectl Shell 访问集群](use-kubectl-and-kubeconfig.md)。 +- **终端远程连接**：你也可以通过在本地桌面上安装 [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)，然后将集群的 kubeconfig 文件复制到本地 `~/.kube/config` 目录来与集群交互。有关更多信息，请参阅[使用 kubectl 和 kubeconfig 文件访问集群](use-kubectl-and-kubeconfig.md)。 + +## Rancher CLI + +你可以下载 Rancher 自己的命令行工具 [Rancher CLI](../../../../reference-guides/cli-with-rancher/cli-with-rancher.md) 来控制你的集群。这个 CLI 工具可以直接与不同的集群和项目进行交互，或者向它们传递 `kubectl` 命令。 + +## Rancher API + +最后，你可以通过 Rancher API 与集群进行交互。在使用 API 之前，你必须先获取 [API 密钥](../../../../reference-guides/user-settings/api-keys.md)。要查看 API 对象的不同资源字段和操作，请打开 API UI（API UI 可以通过单击 Rancher UI 对象的**在 API 中查看**访问）。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md new file mode 100644 index 00000000000..1d76f44b72c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -0,0 +1,79 @@ +--- +title: "Kubernetes 持久存储：卷和存储类" +description: "了解在 Kubernetes 中创建持久存储的两种方法：持久卷和存储类" +--- + + + + + +在部署需要保​​留数据的应用时，你需要创建持久存储。持久存储允许你在运行应用的 pod 之外存储应用数据。即使运行应用的 pod 发生故障，这种存储方式也能让你保留应用数据。 + +本文假设你已了解 Kubernetes 的持久卷、持久卷声明和存储类的概念。如需更多信息，请参阅[存储的工作原理](manage-persistent-storage/about-persistent-storage.md)部分。 + +### 先决条件 + +设置持久存储需要`管理卷`的[角色](../../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色参考)。 + +如果你要为云集群配置存储，则存储和集群主机必须使用相同的云提供商。 + +要使用 Rancher 配置新存储，则必须启用云提供商。有关启用云提供商的详细信息，请参阅[此页面](../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md)。 + +如果要将现有的持久存储连接到集群，则不需要启用云提供商。 + +### 设置现有存储 + +设置现有存储的总体流程如下： + +1. 设置你的持久存储。可以是云存储或你自己的存储。 +2. 添加引用持久存储的持久卷 (PV)。 +3. 添加引用 PV 的持久卷声明 (PVC)。 +4. 将 PVC 挂载为工作负载中的卷。 + +有关详细信息和先决条件，请参阅[此页面](manage-persistent-storage/set-up-existing-storage.md)。 + +### 在 Rancher 中动态配置新存储 + +配置新存储的总体流程如下： + +1. 添加一个 StorageClass 并将它配置为使用你的存储提供商。StorageClass 可以引用云存储或你自己的存储。 +2. 添加引用存储类的持久卷声明 (PVC)。 +3. 将 PVC 挂载为工作负载的卷。 + +有关详细信息和先决条件，请参阅[此页面](manage-persistent-storage/dynamically-provision-new-storage.md)。 + +### Longhorn 存储 + +[Longhorn](https://longhorn.io/) 是一个轻量级、可靠、易用的 Kubernetes 分布式块存储系统。 + +Longhorn 是免费的开源软件。Longhorn 最初由 Rancher Labs 开发，现在正在作为云原生计算基金会的沙盒项目进行开发。它可以通过 Helm、kubectl 或 Rancher UI 安装在任何 Kubernetes 集群上。 + +如果你有块存储池，Longhorn 可以帮助你为 Kubernetes 集群提供持久存储，而无需依赖云提供商。有关 Longhorn 功能的更多信息，请参阅[文档](https://longhorn.io/docs/latest/what-is-longhorn/)。 + +Rancher v2.5 简化了在 Rancher 管理的集群上安装 Longhorn 的过程。详情请参见[本页面](../../../../integrations-in-rancher/longhorn/longhorn.md)。 + +### 配置存储示例 + +我们提供了如何使用 [NFS](../provisioning-storage-examples/nfs-storage.md)， [vSphere](../provisioning-storage-examples/vsphere-storage.md)，和 [Amazon 的 EBS](../provisioning-storage-examples/persistent-storage-in-amazon-ebs.md) 来配置存储的示例。 + +### GlusterFS 卷 + +在将数据存储在 GlusterFS 卷上的集群中，你可能会遇到重启 `kubelet` 后 pod 无法挂载卷的问题。有关避免此情况发生的详细信息，请参阅[此页面](manage-persistent-storage/about-glusterfs-volumes.md)。 + +### iSCSI 卷 + +在将数据存储在 iSCSI 卷上的 [Rancher 启动的 Kubernetes 集群](../../launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)中，你可能会遇到 kubelet 无法自动连接 iSCSI 卷的问题。有关解决此问题的详细信息，请参阅[此页面](manage-persistent-storage/install-iscsi-volumes.md)。 + +### hostPath 卷 + +在创建 hostPath 卷之前，你需要在集群配置中设置 [extra_bind](https://rancher.com/docs/rke/latest/en/config-options/services/services-extras/#extra-binds/)。这会将路径作为卷安装在你的 kubelet 中，可用于工作负载中的 hostPath 卷。 + +### 将 vSphere Cloud Provider 从树内迁移到树外 + +Kubernetes 正在逐渐不在树内维护云提供商。vSphere 有一个树外云提供商，可通过安装 vSphere 云提供商和云存储插件来使用。 + +有关如何从树内 vSphere 云提供商迁移到树外，以及如何在迁移后管理现有虚拟机，请参阅[此页面](../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 + +### 相关链接 + +- [Kubernetes 文档: 存储](https://kubernetes.io/docs/concepts/storage/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/install-cluster-autoscaler/install-cluster-autoscaler.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/install-cluster-autoscaler/install-cluster-autoscaler.md new file mode 100644 index 00000000000..edbea4b5685 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/install-cluster-autoscaler/install-cluster-autoscaler.md @@ -0,0 +1,28 @@ +--- +title: Cluster Autoscaler +--- + + + + + +在本文中，你将学习如何使用 AWS EC2 Auto Scaling 组在 Rancher 自定义集群上安装和使用 [Kubernetes cluster-autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/)。 + +Cluster Autoscaler 是一个自动调整 Kubernetes 集群大小的工具。该工具在满足以下条件之一时能自动调整集群大小： + +* 集群中有 Pod 因资源不足而无法运行。 +* 集群中有一些节点长时间未得到充分利用，而且它们的 Pod 可以放到其他现有节点上。 + +为防止你的 pod 被驱逐，请在你的 pod 规范中设置 `priorityClassName: system-cluster-critical` 属性。 + +Cluster Autoscaler 运行在 Kubernetes master 节点上。它可以在 `kube-system` 命名空间中运行。Cluster Autoscaler 不会缩减运行非镜像 `kube-system` pod 的节点。 + +你可以在 worker 节点上运行 Cluster Autoscaler 的自定义 deployment，但需要小心以保证 Cluster Autoscaler 能正常运行。 + +## 云提供商 + +Cluster Autoscaler 为不同的云提供商提供支持。有关详细信息，请参见 [Cluster Autoscaler 支持的云提供商](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#deployment)。 + +### 在亚马逊云提供商上设置 Cluster Autoscaler + +有关在 Amazon 上运行 Cluster Autoscaler 的详细信息，请参阅[此页面](use-aws-ec2-auto-scaling-groups.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/manage-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/manage-clusters.md new file mode 100644 index 00000000000..f7d9bb6b990 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/manage-clusters.md @@ -0,0 +1,34 @@ +--- +title: 集群管理 +--- + + + + + +在 Rancher 中配置集群后，你可以开始使用强大的 Kubernetes 功能在开发、测试或生产环境中部署和扩展容器化应用。 + +:::note + +本节默认你已对 Docker 和 Kubernetes 有一定的了解。如果你需要了解 Kubernetes 组件如何协作，请参见 [Kubernetes 概念](../../../reference-guides/kubernetes-concepts.md)页面。 + +::: + +## 在 Rancher 中管理集群 + +将集群[配置到 Rancher](../kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md) 之后，[集群所有者](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles)需要管理这些集群。管理集群的选项如下： + +import ClusterCapabilitiesTable from '../../../shared-files/_cluster-capabilities-table.md'; + + + +Rancher 包含 Kubernetes 中未包含的各种工具来协助你进行 DevOps 操作。Rancher 可以与外部服务集成，让你的集群更高效地运行。工具分为以下几类： + +- 告警 +- Notifiers +- Logging +- Monitoring +- Istio 服务网格 +- OPA Gatekeeper + +你可以通过 **Apps** 来安装工具。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/provisioning-storage-examples.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/provisioning-storage-examples.md new file mode 100644 index 00000000000..2509db22979 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/provisioning-storage-examples.md @@ -0,0 +1,15 @@ +--- +title: 配置存储示例 +--- + + + + + +Rancher 通过各种卷插件支持持久化存储。不过，在使用这些插件将持久存储绑定到工作负载之前，您必须先配置存储本身，无论是服务提供商提供的基于云的解决方案，还是您自己管理的内部部署解决方案。 + +Rancher 提供了有关如何配置一些常用存储方法的文档： + +- [NFS](nfs-storage.md) +- [vSphere](vsphere-storage.md) +- [EBS](persistent-storage-in-amazon-ebs.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/shared-files/_cluster-capabilities-table.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/shared-files/_cluster-capabilities-table.md index c418e1eaf54..c4b68aa5d8c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/shared-files/_cluster-capabilities-table.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/shared-files/_cluster-capabilities-table.md @@ -2,13 +2,13 @@ | --- | --- | ---| ---|----| | [使用 kubectl 和 kubeconfig 文件来访问集群](../how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md) | ✓ | ✓ | ✓ | ✓ | | [管理集群成员](../how-to-guides/new-user-guides/manage-clusters/access-clusters/add-users-to-clusters.md) | ✓ | ✓ | ✓ | ✓ | -| [编辑和升级集群](../pages-for-subheaders/cluster-configuration.md) | ✓ | ✓ | ✓ | ✓2 | +| [编辑和升级集群](../reference-guides/cluster-configuration/cluster-configuration.md) | ✓ | ✓ | ✓ | ✓2 | | [管理节点](../how-to-guides/new-user-guides/manage-clusters/nodes-and-node-pools.md) | ✓ | ✓ | ✓ | ✓3 | -| [管理持久卷和存储类](../pages-for-subheaders/create-kubernetes-persistent-storage.md) | ✓ | ✓ | ✓ | ✓ | +| [管理持久卷和存储类](../how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md) | ✓ | ✓ | ✓ | ✓ | | [管理项目、命名空间和工作负载](../how-to-guides/new-user-guides/manage-clusters/projects-and-namespaces.md) | ✓ | ✓ | ✓ | ✓ | -| [使用应用目录](../pages-for-subheaders/helm-charts-in-rancher.md) | ✓ | ✓ | ✓ | ✓ | -| 配置工具（[Alerts、Notifiers、Monitoring](../pages-for-subheaders/monitoring-and-alerting.md)、[Logging](../pages-for-subheaders/logging.md) 和 [Istio](../pages-for-subheaders/istio.md)） | ✓ | ✓ | ✓ | ✓ | -| [运行安全扫描](../pages-for-subheaders/cis-scan-guides.md) | ✓ | ✓ | ✓ | ✓ | +| [使用应用目录](../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md) | ✓ | ✓ | ✓ | ✓ | +| 配置工具（[Alerts、Notifiers、Monitoring](../integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md)、[Logging](../integrations-in-rancher/logging/logging.md) 和 [Istio](../integrations-in-rancher/istio/istio.md)） | ✓ | ✓ | ✓ | ✓ | +| [运行安全扫描](../how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md) | ✓ | ✓ | ✓ | ✓ | | [轮换证书](../how-to-guides/new-user-guides/manage-clusters/rotate-certificates.md) | ✓ | ✓ | | | | [备份](../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher-launched-kubernetes-clusters.md)和[恢复](../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher-launched-kubernetes-clusters-from-backup.md) Rancher 启动的集群 | ✓ | ✓ | | ✓4 | | [在 Rancher 无法访问集群时清理 Kubernetes 组件](../how-to-guides/new-user-guides/manage-clusters/clean-cluster-nodes.md) | ✓ | | | | From 957bd368a75fd38050be9e96a6aa27237e07b6bc Mon Sep 17 00:00:00 2001 From: Jacie Date: Mon, 15 Apr 2024 21:04:34 +0800 Subject: [PATCH 46/67] update missing Chinese translation for latest and v2.8 (#1234) - Cluster API - Deprecated features --- .../current.json | 88 +++---- .../current/faq/deprecated-features.md | 26 +++ .../cluster-api/cluster-api.md | 14 ++ .../cluster-api/overview.md | 221 ++++++++++++++++++ .../version-2.8.json | 88 +++---- .../version-2.8/faq/deprecated-features.md | 26 +++ .../cluster-api/cluster-api.md | 14 ++ .../cluster-api/overview.md | 221 ++++++++++++++++++ 8 files changed, 614 insertions(+), 84 deletions(-) create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/faq/deprecated-features.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/cluster-api.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/faq/deprecated-features.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/overview.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current.json b/i18n/zh/docusaurus-plugin-content-docs/current.json index 5a6ef279517..f1e762f44f9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current.json +++ b/i18n/zh/docusaurus-plugin-content-docs/current.json @@ -11,13 +11,13 @@ "message": "快速入门指南", "description": "The label for category Quick Start Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Deploy Rancher": { + "sidebar.tutorialSidebar.category.Deploying Rancher Server": { "message": "部署 Rancher", - "description": "The label for category Deploy Rancher in sidebar tutorialSidebar" + "description": "The label for category Deploying Rancher Server in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Deploy Workloads": { + "sidebar.tutorialSidebar.category.Deploying Workloads": { "message": "部署工作负载", - "description": "The label for category Deploy Workloads in sidebar tutorialSidebar" + "description": "The label for category Deploying Workloads in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Installation and Upgrade": { "message": "安装和升级", @@ -95,9 +95,9 @@ "message": "关于 RKE1 模板", "description": "The label for category About RKE1 Templates in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Manage Clusters": { - "message": "管理集群", - "description": "The label for category Manage Clusters in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Cluster Administration": { + "message": "集群管理", + "description": "The label for category Cluster Administration in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Access Clusters": { "message": "访问集群", @@ -115,9 +115,9 @@ "message": "配置存储示例", "description": "The label for category Provisioning Storage Examples in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Kubernetes Cluster Setup": { + "sidebar.tutorialSidebar.category.Setting up a Kubernetes Cluster for Rancher Server": { "message": "Kubernetes 集群设置", - "description": "The label for category Kubernetes Cluster Setup in sidebar tutorialSidebar" + "description": "The label for category Setting up a Kubernetes Cluster for Rancher Server in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Infrastructure Setup": { "message": "基础设施设置", @@ -156,16 +156,16 @@ "description": "The label for category Launch Kubernetes with Rancher in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Launching New Nodes in an Infra Provider": { - "message": "在基础设施提供商中启动新节点", + "message": "在基础设施提供商中使用新节点", "description": "The label for category Launching New Nodes in an Infra Provider in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.vSphere": { - "message": "vSphere", - "description": "The label for category vSphere in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Creating a vSphere Cluster": { + "message": "创建 vSphere 集群", + "description": "The label for category Creating a vSphere Cluster in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Nutanix": { - "message": "Nutanix", - "description": "The label for category Nutanix in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Creating a Nutanix AOS Cluster": { + "message": "创建 Nutanix AOS 集群", + "description": "The label for category Creating a Nutanix AOS Cluster in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Kubernetes Resources Setup": { "message": "Kubernetes 资源设置", @@ -183,9 +183,9 @@ "message": "负载均衡和 Ingress Controller", "description": "The label for category Load Balancer and Ingress Controller in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Helm Charts in Rancher": { - "message": "Rancher 中的 Helm Chart", - "description": "The label for category Helm Charts in Rancher in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Helm Charts and Apps": { + "message": "Helm Charts 和 Apps", + "description": "The label for category Helm Charts and Apps in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Deploy Apps Across Clusters": { "message": "跨集群部署应用", @@ -199,13 +199,13 @@ "message": "高级用户指南", "description": "The label for category Advanced User Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Manage Projects": { - "message": "管理项目", - "description": "The label for category Manage Projects in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Project Administration": { + "message": "项目管理", + "description": "The label for category Project Administration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Manage Project Resource Quotas": { - "message": "管理项目资源配额", - "description": "The label for category Manage Project Resource Quotas in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Project Resource Quotas": { + "message": "项目资源配额", + "description": "The label for category Project Resource Quotas in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Monitoring/Alerting Guides": { "message": "Monitoring/Alerting 指南", @@ -215,33 +215,33 @@ "message": "Prometheus Federator 指南", "description": "The label for category Prometheus Federator Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Monitoring V2 Configuration Guides": { - "message": "Monitoring V2 配置指南", - "description": "The label for category Monitoring V2 Configuration Guides in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Monitoring Configuration Guides": { + "message": "Monitoring 配置指南", + "description": "The label for category Monitoring Configuration Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Advanced Configuration": { "message": "高级配置", "description": "The label for category Advanced Configuration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Istio Setup Guide": { + "sidebar.tutorialSidebar.category.Istio Setup Guides": { "message": "Istio 设置指南", - "description": "The label for category Istio Setup Guide in sidebar tutorialSidebar" + "description": "The label for category Istio Setup Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.CIS Scan Guides": { "message": "CIS 扫描指南", "description": "The label for category CIS Scan Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Enable Experimental Features": { + "sidebar.tutorialSidebar.category.Enabling Experimental Features": { "message": "启用实验功能", - "description": "The label for category Enable Experimental Features in sidebar tutorialSidebar" + "description": "The label for category Enabling Experimental Features in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Reference Guides": { "message": "参考指南", "description": "The label for category Reference Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Best Practices": { + "sidebar.tutorialSidebar.category.Best Practice Guides": { "message": "最佳实践", - "description": "The label for category Best Practices in sidebar tutorialSidebar" + "description": "The label for category Best Practice Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Rancher Server": { "message": "Rancher Server", @@ -263,9 +263,9 @@ "message": "Rancher Server 配置", "description": "The label for category Rancher Server Configuration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.GKE Cluster Configuration": { + "sidebar.tutorialSidebar.category.GKE Cluster Configuration Reference": { "message": "GKE 集群配置", - "description": "The label for category GKE Cluster Configuration in sidebar tutorialSidebar" + "description": "The label for category GKE Cluster Configuration Reference in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Use Existing Nodes": { "message": "使用现有节点", @@ -291,9 +291,9 @@ "message": "备份和恢复配置", "description": "The label for category Backup & Restore Configuration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Monitoring V2 Configuration": { - "message": "Monitoring V2 配置", - "description": "The label for category Monitoring V2 Configuration in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Monitoring Configuration Reference": { + "message": "Monitoring 配置", + "description": "The label for category Monitoring Configuration Reference in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Prometheus Federator": { "message": "Prometheus Federator", @@ -311,9 +311,9 @@ "message": "关于 API", "description": "The label for category About the API in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Rancher Security": { - "message": "Rancher 安全", - "description": "The label for category Rancher Security in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Rancher Security Guides": { + "message": "Rancher 安全指南", + "description": "The label for category Rancher Security Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.SELinux RPM": { "message": "SELinux RPM", @@ -406,5 +406,9 @@ "sidebar.tutorialSidebar.category.Container Security with Neuvector": { "message": "使用 NeuVector 实现容器安全", "description": "The label for category Container Security with Neuvector in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Cluster API (CAPI) with Rancher Turtles": { + "message": "Cluster API (CAPI) 与 Rancher Turtles", + "description": "The label for category Cluster API (CAPI) with Rancher Turtles in sidebar tutorialSidebar" } } diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/faq/deprecated-features.md b/i18n/zh/docusaurus-plugin-content-docs/current/faq/deprecated-features.md new file mode 100644 index 00000000000..b594bd0a42b --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/faq/deprecated-features.md @@ -0,0 +1,26 @@ +--- +title: Rancher 中已弃用的功能 +--- + + + + + +### Rancher 的弃用策略是什么？ + +我们已经在支持的[服务条款](https://rancher.com/support-maintenance-terms)中发布了官方的弃用策略。 + +### 在哪里可以了解 Rancher 中已弃用哪些功能？ + +Rancher 将在 GitHub 上发布的 Rancher 的[发版说明](https://github.com/rancher/rancher/releases)中发布已弃用的功能。有关已弃用的功能，请参阅以下的补丁版本： + +| Patch 版本 | 发布时间 | +| --------------------------------------------------------------- | ------------------ | +| [2.8.3](https://github.com/rancher/rancher/releases/tag/v2.8.3) | 2024 年 3 月 28 日 | +| [2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2) | 2024 年 2 月 8 日 | +| [2.8.1](https://github.com/rancher/rancher/releases/tag/v2.8.1) | 2024 年 1 月 22 日 | +| [2.8.0](https://github.com/rancher/rancher/releases/tag/v2.8.0) | 2023 年 12 月 6 日 | + +### 当一个功能被标记为弃用我可以得到什么样的预期？ + +当功能被标记为“已弃用”时，它依然可用并得到支持，允许按照常规的流程进行升级。一旦升级完成，用户/管理员应开始计划在升级到标记为已移除的版本之前放弃使用已弃用的功能。对于新的部署，建议不要使用已弃用的功能。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/cluster-api.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/cluster-api.md new file mode 100644 index 00000000000..aa10ce27651 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/cluster-api.md @@ -0,0 +1,14 @@ +--- +title: Cluster API (CAPI) 与 Rancher Turtles +--- + + + + + +[Rancher Turtles](https://turtles.docs.rancher.com/) 是一个 [Rancher 扩展](../rancher-extensions.md)，通过提供 Cluster API (CAPI) 和 Rancher 之间的集成来管理配置的 Kubernetes 集群的生命周期。使用 Rancher Turtles，你可以： + +- 通过在 CAPI 配置的集群中安装 Rancher Cluster Agent，将 CAPI 集群导入 Rancher。 +- 配置 [CAPI Operator](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values)。 + +[概述](./overview.md)部分介绍了安装选项、Rancher Turtles 架构和简要 Demo。有关详细信息，请参阅 [Rancher Turtles 文档](https://turtles.docs.rancher.com/)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/overview.md new file mode 100644 index 00000000000..2981c2199ef --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/overview.md @@ -0,0 +1,221 @@ +--- +title: 概述 +--- + + + + + +## 架构图 + +下面是 Rancher Turtles 的关键组件及其与 Rancher 和 Rancher Cluster Agent 的关系的架构图，了解这些组件对于深入了解 Rancher 如何利用 CAPI operator 进行集群管理至关重要。 + +![overview](/img/30000ft_view.png) + +## 先决条件 + +在 Rancher 环境中安装 Rancher Turtles 之前，你必须禁用 Rancher 的 `embedded-cluster-api` 功能。这还包括清理 Rancher 专用的 webhook，否则这些 webhook 将与 CAPI 的 webhook 冲突。 + +为了简化 Rancher 安装 Rancher Turtles 的设置，官方的 Rancher Turtles Helm chart 包含一个删除以下内容的 `pre-install` hook： + +- 禁用 Rancher 中的 `embedded-cluster-api` 功能。 +- 删除不再需要的 `mutating-webhook-configuration` 和 `validating-webhook-configuration` webhook。 + +## 安装 Rancher Turtles Operator + +你可以通过 Rancher UI 或使用 Helm 安装 Rancher Turtles operator。对于大多数环境推荐使用第一种方法。 + +:::caution + +如果你的集群中已经安装了 Cluster API (CAPI) Operator，你必须使用[手动 Helm 安装方法](#通过-helm-安装)。 + +::: + +### 通过 Rancher UI 安装 + +通过 Rancher UI 添加 Turtles 仓库，Rancher 可以处理 CAPI 扩展的安装和配置。 + +1. 点击 **☰**。在左侧导航栏**浏览集群**的菜单下，选择 **local**。 +1. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 +1. 点击 **Create** 创建一个新的仓库。 +1. 输入以下信息： + - **Name**: turtles + - **Index URL**: https://rancher.github.io/turtles +1. 等待新的仓库状态更新为 `Active`。 +1. 在左侧导航菜单中，点击 **Apps > Charts**。 +1. 在搜索过滤器中输入 "turtles" 来查找 Turtles chart。 +1. 点击 **Rancher Turtles - the Cluster API Extension**。 +1. 点击 **Install > Next > Install**. + +此过程使用 Helm chart 的默认值，这些值适用于大部分的安装场景。如果你的配置需要覆盖其中一些默认值，你可以在安装期间通过 Rancher UI 指定这些值，也可以通过 [Helm 手动安装 Chart](#通过-helm-安装)。有关可用的 values 设置的详细信息，请参阅 Rancher Turtles 的 [Helm chart 参考指南](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values)。 + +安装可能需要几分钟时间，安装完成后，你可以在集群中看到以下新部署： + +- `rancher-turtles-system/rancher-turtles-controller-manager` +- `rancher-turtles-system/rancher-turtles-cluster-api-operator` +- `capi-system/capi-controller-manager` + +#### Demo + +这个 demo 演示了如何使用 Rancher UI 安装 Rancher Turtles、创建/导入一个 CAPI 集群，以及在集群上安装监控： + + + +### 通过 Helm 安装 + +通过 Helm 安装 Rancher Turtles 有两种方法，这取决于你是否将 CAPI operator 作为依赖项包含其中： + +- [使用 CAPI Operator 作为依赖项安装 Rancher Turtles](#使用-cluster-api-capi-operator-作为-helm-依赖项安装-rancher-turtles)。 +- [安装没有 CAPI Operator 的 Rancher Turtles](#不使用-cluster-api-capi-operator-作为-helm-依赖安装-rancher-turtles)。 + +安装 Rancher Turtles 需要 CAPI Operator。你可以选择自己处理此依赖项，还是让 Rancher Turtles Helm chart 替你管理它。[使用 CAPI Operator 作为依赖项安装 Rancher Turtles](#使用-cluster-api-capi-operator-作为-helm-依赖项安装-rancher-turtles) 更简单，但是你的最佳选择取决于你的具体配置。 + +CAPI Operator 允许使用声明式方法处理 CAPI provider 的生命周期，扩展了 `clusterctl` 的能力。如果你想了解更多相关内容，可以参考 [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/)。 + +#### 使用 `Cluster API (CAPI) Operator` 作为 Helm 依赖项安装 Rancher Turtles + +1. 添加包含 `rancher-turtles` chart 的 Helm 仓库作为安装的第一步： + +```bash +helm repo add turtles https://rancher.github.io/turtles +helm repo update +``` + +2. 如前面所述，安装 Rancher Turtles 需要 [CAPI Operator](https://github.com/kubernetes-sigs/cluster-api-operator)。Helm chart 可以使用一组最少的参数自动安装： + +```bash +helm install rancher-turtles turtles/rancher-turtles --version \ + -n rancher-turtles-system \ + --dependency-update \ + --create-namespace --wait \ + --timeout 180s +``` + +3. 此操作可能需要几分钟时间，完成后，你可以查看下面列出的已安装的控制器： + +- `rancher-turtles-controller` +- `capi-operator` + +:::note + +- 如果集群中已经有可用的 `cert-manager`，请在安装时通过以下参数将 Rancher Turtles 的此项依赖禁用掉，来阻止冲突： + `--set cluster-api-operator.cert-manager.enabled=false` +- 有关 Rancher Turtles 的版本列表，请参阅 [Turtles 发布页面](https://github.com/rancher/turtles/releases)。 + +::: + +这是最基本的推荐配置，用于管理在核心提供商的命名空间中创建包含所需 CAPI 功能标志（已启用 `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` 和 `EXP_MACHINE_POOL` ）的 secret。要启用其他 CAPI 功能需要启用上述功能标志。 + +如果你需要覆盖默认的行为并使用现有 secret 或添加自定义环境变量，你可以将 secret 名称通过 Helm 参数传入。在这种情况下，你作为负责管理 secret 创建和内容的用户，需要启用的最少特性包括：`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` 和 `EXP_MACHINE_POOL`。 + +```bash +helm install ... + # Passing secret name and namespace for additional environment variables + --set cluster-api-operator.cluster-api.configSecret.name= +``` + +以下是一个用户管理 secret `cluster-api-operator.cluster-api.configSecret.name=variables` 的示例，其中设置了 `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` 和 `EXP_MACHINE_POOL` 功能以及一个额外的自定义变量： + +```yaml title="secret.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: variables + namespace: rancher-turtles-system +type: Opaque +stringData: + CLUSTER_TOPOLOGY: "true" + EXP_CLUSTER_RESOURCE_SET: "true" + EXP_MACHINE_POOL: "true" + CUSTOM_ENV_VAR: "false" +``` + +:::info + +有关 chart 支持的 values 及其用法的详细信息，请参阅 [Helm chart 选项](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) + +::: + +#### 不使用 `Cluster API (CAPI) Operator` 作为 Helm 依赖安装 Rancher Turtles + +:::note + +请记住，如果使用此安装选项，你必须自行管理 CAPI Operator 的安装。你可以参照 Rancher Turtles 文档中的 [CAPI Operator 指南](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) + +::: + +1. 添加包含 `rancher-turtles` chart 的 Helm 仓库作为安装的第一步： + +```bash +helm repo add turtles https://rancher.github.io/turtles +helm repo update +``` + +2. 将 chart 安装到 `rancher-turtles-system` 命名空间： + +```bash +helm install rancher-turtles turtles/rancher-turtles --version + -n rancher-turtles-system + --set cluster-api-operator.enabled=false + --set cluster-api-operator.cluster-api.enabled=false + --create-namespace --wait + --dependency-update +``` + +前面的命令告诉 Helm 忽略将 `cluster-api-operator` 作为依赖项安装。 + +3. 此操作可能需要几分钟，完成后你可以查看下面列出的已安装的控制器： + +- `rancher-turtles-controller` + +## 卸载 Rancher Turtles + +:::caution + +在 Rancher 环境中安装 Rancher Turtles 时，Rancher Turtles 默认会启用 CAPI Operator 清理。这包括清理 CAPI Operator 特定的 webhook 和 deployments，否则会导致 Rancher 配置出现问题。 + +为了简化 Rancher Turtles 卸载（通过 Rancher 或 Helm 命令），官方的 Rancher Turtles Helm chart 包含了一个删除以下内容的 `post-delete` hook： + +- 删除不再需要的 `mutating-webhook-configuration` 和 `validating-webhook-configuration` webhook。 +- 删除不再需要的 CAPI `deployments`。 + +::: + +卸载 Rancher Turtles： + +```bash +helm uninstall -n rancher-turtles-system rancher-turtles --cascade foreground --wait +``` + +这可能需要几分钟才能完成。 + +:::note + +请记住，如果你在安装时使用了不同的名称或不同的命名空间，你需要针对你的特定配置自定义卸载命令。 + +::: + +卸载 Rancher Turtles 后， Rancher 的 `embedded-cluster-api` 功能必须重新启用。 + +1. 创建一个 `feature.yaml` 文件，将 `embedded-cluster-api` 设置为 true： + +```yaml title="feature.yaml" +apiVersion: management.cattle.io/v3 +kind: Feature +metadata: + name: embedded-cluster-api +spec: + value: true +``` + +2. 使用 `kubectl` 将 `feature.yaml` 文件应用到集群： + +```bash +kubectl apply -f feature.yaml +``` + +## 安全 + +[SLSA](https://slsa.dev/spec/v1.0/about) 是一套由行业共识制定的可逐步采用的供应链安全指南。SLSA 制定的规范对软件生产者和消费者都很有用：生产者可以遵循 SLSA 的指导方针，使他们的软件供应链更加安全，消费者可以使用 SLSA 来决定是否信任软件包。 + +Rancher Turtles 满足 [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) 对适当的构建平台、一致的构建过程和来源分布的要求。更多信息请参阅 [Rancher Turtles 安全](https://turtles.docs.rancher.com/docs/security/slsa)文档。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8.json b/i18n/zh/docusaurus-plugin-content-docs/version-2.8.json index b4b95388418..1562e2d0991 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8.json +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8.json @@ -11,13 +11,13 @@ "message": "快速入门指南", "description": "The label for category Quick Start Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Deploy Rancher": { + "sidebar.tutorialSidebar.category.Deploying Rancher Server": { "message": "部署 Rancher", - "description": "The label for category Deploy Rancher in sidebar tutorialSidebar" + "description": "The label for category Deploying Rancher Server in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Deploy Workloads": { + "sidebar.tutorialSidebar.category.Deploying Workloads": { "message": "部署工作负载", - "description": "The label for category Deploy Workloads in sidebar tutorialSidebar" + "description": "The label for category Deploying Workloads in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Installation and Upgrade": { "message": "安装和升级", @@ -95,9 +95,9 @@ "message": "关于 RKE1 模板", "description": "The label for category About RKE1 Templates in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Manage Clusters": { - "message": "管理集群", - "description": "The label for category Manage Clusters in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Cluster Administration": { + "message": "集群管理", + "description": "The label for category Cluster Administration in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Access Clusters": { "message": "访问集群", @@ -115,9 +115,9 @@ "message": "配置存储示例", "description": "The label for category Provisioning Storage Examples in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Kubernetes Cluster Setup": { + "sidebar.tutorialSidebar.category.Setting up a Kubernetes Cluster for Rancher Server": { "message": "Kubernetes 集群设置", - "description": "The label for category Kubernetes Cluster Setup in sidebar tutorialSidebar" + "description": "The label for category Setting up a Kubernetes Cluster for Rancher Server in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Infrastructure Setup": { "message": "基础设施设置", @@ -147,7 +147,7 @@ "message": "设置 Cloud Provider", "description": "The label for category Setting up Cloud Providers in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Launch Kubernetes with Rancher": { + "sidebar.tutorialSidebar.category.Launching Kubernetes with Rancher": { "message": "使用 Rancher 启动 Kubernetes", "description": "The label for category Launch Kubernetes with Rancher in sidebar tutorialSidebar" }, @@ -159,13 +159,13 @@ "message": "在基础设施提供商中使用新节点", "description": "The label for category Launching New Nodes in an Infra Provider in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.vSphere": { - "message": "vSphere", - "description": "The label for category vSphere in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Creating a vSphere Cluster": { + "message": "创建 vSphere 集群", + "description": "The label for category Creating a vSphere Cluster in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Nutanix": { - "message": "Nutanix", - "description": "The label for category Nutanix in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Creating a Nutanix AOS Cluster": { + "message": "创建 Nutanix AOS 集群", + "description": "The label for category Creating a Nutanix AOS Cluster in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Kubernetes Resources Setup": { "message": "Kubernetes 资源设置", @@ -183,9 +183,9 @@ "message": "负载均衡和 Ingress Controller", "description": "The label for category Load Balancer and Ingress Controller in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Helm Charts in Rancher": { - "message": "Rancher 中的 Helm Chart", - "description": "The label for category Helm Charts in Rancher in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Helm Charts and Apps": { + "message": "Helm Charts 和 Apps", + "description": "The label for category Helm Charts and Apps in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Deploy Apps Across Clusters": { "message": "跨集群部署应用", @@ -199,13 +199,13 @@ "message": "高级用户指南", "description": "The label for category Advanced User Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Manage Projects": { - "message": "管理项目", - "description": "The label for category Manage Projects in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Project Administration": { + "message": "项目管理", + "description": "The label for category Project Administration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Manage Project Resource Quotas": { - "message": "管理项目资源配额", - "description": "The label for category Manage Project Resource Quotas in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Project Resource Quotas": { + "message": "项目资源配额", + "description": "The label for category Project Resource Quotas in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Monitoring/Alerting Guides": { "message": "Monitoring/Alerting 指南", @@ -215,33 +215,33 @@ "message": "Prometheus Federator 指南", "description": "The label for category Prometheus Federator Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Monitoring V2 Configuration Guides": { - "message": "Monitoring V2 配置指南", - "description": "The label for category Monitoring V2 Configuration Guides in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Monitoring Configuration Guides": { + "message": "Monitoring 配置指南", + "description": "The label for category Monitoring Configuration Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Advanced Configuration": { "message": "高级配置", "description": "The label for category Advanced Configuration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Istio Setup Guide": { + "sidebar.tutorialSidebar.category.Istio Setup Guides": { "message": "Istio 设置指南", - "description": "The label for category Istio Setup Guide in sidebar tutorialSidebar" + "description": "The label for category Istio Setup Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.CIS Scan Guides": { "message": "CIS 扫描指南", "description": "The label for category CIS Scan Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Enable Experimental Features": { + "sidebar.tutorialSidebar.category.Enabling Experimental Features": { "message": "启用实验功能", - "description": "The label for category Enable Experimental Features in sidebar tutorialSidebar" + "description": "The label for category Enabling Experimental Features in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Reference Guides": { "message": "参考指南", "description": "The label for category Reference Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Best Practices": { + "sidebar.tutorialSidebar.category.Best Practice Guides": { "message": "最佳实践", - "description": "The label for category Best Practices in sidebar tutorialSidebar" + "description": "The label for category Best Practice Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Rancher Server": { "message": "Rancher Server", @@ -263,9 +263,9 @@ "message": "Rancher Server 配置", "description": "The label for category Rancher Server Configuration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.GKE Cluster Configuration": { + "sidebar.tutorialSidebar.category.GKE Cluster Configuration Reference": { "message": "GKE 集群配置", - "description": "The label for category GKE Cluster Configuration in sidebar tutorialSidebar" + "description": "The label for category GKE Cluster Configuration Reference in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Use Existing Nodes": { "message": "使用现有节点", @@ -291,9 +291,9 @@ "message": "备份和恢复配置", "description": "The label for category Backup & Restore Configuration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Monitoring V2 Configuration": { - "message": "Monitoring V2 配置", - "description": "The label for category Monitoring V2 Configuration in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Monitoring Configuration Reference": { + "message": "Monitoring 配置", + "description": "The label for category Monitoring Configuration Reference in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Prometheus Federator": { "message": "Prometheus Federator", @@ -311,9 +311,9 @@ "message": "关于 API", "description": "The label for category About the API in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Rancher Security": { - "message": "Rancher 安全", - "description": "The label for category Rancher Security in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Rancher Security Guides": { + "message": "Rancher 安全指南", + "description": "The label for category Rancher Security Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.SELinux RPM": { "message": "SELinux RPM", @@ -406,6 +406,10 @@ "sidebar.tutorialSidebar.category.Container Security with Neuvector": { "message": "使用 NeuVector 实现容器安全", "description": "The label for category Container Security with Neuvector in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Cluster API (CAPI) with Rancher Turtles": { + "message": "Cluster API (CAPI) 与 Rancher Turtles", + "description": "The label for category Cluster API (CAPI) with Rancher Turtles in sidebar tutorialSidebar" } } \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/faq/deprecated-features.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/faq/deprecated-features.md new file mode 100644 index 00000000000..b594bd0a42b --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/faq/deprecated-features.md @@ -0,0 +1,26 @@ +--- +title: Rancher 中已弃用的功能 +--- + + + + + +### Rancher 的弃用策略是什么？ + +我们已经在支持的[服务条款](https://rancher.com/support-maintenance-terms)中发布了官方的弃用策略。 + +### 在哪里可以了解 Rancher 中已弃用哪些功能？ + +Rancher 将在 GitHub 上发布的 Rancher 的[发版说明](https://github.com/rancher/rancher/releases)中发布已弃用的功能。有关已弃用的功能，请参阅以下的补丁版本： + +| Patch 版本 | 发布时间 | +| --------------------------------------------------------------- | ------------------ | +| [2.8.3](https://github.com/rancher/rancher/releases/tag/v2.8.3) | 2024 年 3 月 28 日 | +| [2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2) | 2024 年 2 月 8 日 | +| [2.8.1](https://github.com/rancher/rancher/releases/tag/v2.8.1) | 2024 年 1 月 22 日 | +| [2.8.0](https://github.com/rancher/rancher/releases/tag/v2.8.0) | 2023 年 12 月 6 日 | + +### 当一个功能被标记为弃用我可以得到什么样的预期？ + +当功能被标记为“已弃用”时，它依然可用并得到支持，允许按照常规的流程进行升级。一旦升级完成，用户/管理员应开始计划在升级到标记为已移除的版本之前放弃使用已弃用的功能。对于新的部署，建议不要使用已弃用的功能。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md new file mode 100644 index 00000000000..aa10ce27651 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md @@ -0,0 +1,14 @@ +--- +title: Cluster API (CAPI) 与 Rancher Turtles +--- + + + + + +[Rancher Turtles](https://turtles.docs.rancher.com/) 是一个 [Rancher 扩展](../rancher-extensions.md)，通过提供 Cluster API (CAPI) 和 Rancher 之间的集成来管理配置的 Kubernetes 集群的生命周期。使用 Rancher Turtles，你可以： + +- 通过在 CAPI 配置的集群中安装 Rancher Cluster Agent，将 CAPI 集群导入 Rancher。 +- 配置 [CAPI Operator](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values)。 + +[概述](./overview.md)部分介绍了安装选项、Rancher Turtles 架构和简要 Demo。有关详细信息，请参阅 [Rancher Turtles 文档](https://turtles.docs.rancher.com/)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/overview.md new file mode 100644 index 00000000000..2981c2199ef --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -0,0 +1,221 @@ +--- +title: 概述 +--- + + + + + +## 架构图 + +下面是 Rancher Turtles 的关键组件及其与 Rancher 和 Rancher Cluster Agent 的关系的架构图，了解这些组件对于深入了解 Rancher 如何利用 CAPI operator 进行集群管理至关重要。 + +![overview](/img/30000ft_view.png) + +## 先决条件 + +在 Rancher 环境中安装 Rancher Turtles 之前，你必须禁用 Rancher 的 `embedded-cluster-api` 功能。这还包括清理 Rancher 专用的 webhook，否则这些 webhook 将与 CAPI 的 webhook 冲突。 + +为了简化 Rancher 安装 Rancher Turtles 的设置，官方的 Rancher Turtles Helm chart 包含一个删除以下内容的 `pre-install` hook： + +- 禁用 Rancher 中的 `embedded-cluster-api` 功能。 +- 删除不再需要的 `mutating-webhook-configuration` 和 `validating-webhook-configuration` webhook。 + +## 安装 Rancher Turtles Operator + +你可以通过 Rancher UI 或使用 Helm 安装 Rancher Turtles operator。对于大多数环境推荐使用第一种方法。 + +:::caution + +如果你的集群中已经安装了 Cluster API (CAPI) Operator，你必须使用[手动 Helm 安装方法](#通过-helm-安装)。 + +::: + +### 通过 Rancher UI 安装 + +通过 Rancher UI 添加 Turtles 仓库，Rancher 可以处理 CAPI 扩展的安装和配置。 + +1. 点击 **☰**。在左侧导航栏**浏览集群**的菜单下，选择 **local**。 +1. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 +1. 点击 **Create** 创建一个新的仓库。 +1. 输入以下信息： + - **Name**: turtles + - **Index URL**: https://rancher.github.io/turtles +1. 等待新的仓库状态更新为 `Active`。 +1. 在左侧导航菜单中，点击 **Apps > Charts**。 +1. 在搜索过滤器中输入 "turtles" 来查找 Turtles chart。 +1. 点击 **Rancher Turtles - the Cluster API Extension**。 +1. 点击 **Install > Next > Install**. + +此过程使用 Helm chart 的默认值，这些值适用于大部分的安装场景。如果你的配置需要覆盖其中一些默认值，你可以在安装期间通过 Rancher UI 指定这些值，也可以通过 [Helm 手动安装 Chart](#通过-helm-安装)。有关可用的 values 设置的详细信息，请参阅 Rancher Turtles 的 [Helm chart 参考指南](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values)。 + +安装可能需要几分钟时间，安装完成后，你可以在集群中看到以下新部署： + +- `rancher-turtles-system/rancher-turtles-controller-manager` +- `rancher-turtles-system/rancher-turtles-cluster-api-operator` +- `capi-system/capi-controller-manager` + +#### Demo + +这个 demo 演示了如何使用 Rancher UI 安装 Rancher Turtles、创建/导入一个 CAPI 集群，以及在集群上安装监控： + + + +### 通过 Helm 安装 + +通过 Helm 安装 Rancher Turtles 有两种方法，这取决于你是否将 CAPI operator 作为依赖项包含其中： + +- [使用 CAPI Operator 作为依赖项安装 Rancher Turtles](#使用-cluster-api-capi-operator-作为-helm-依赖项安装-rancher-turtles)。 +- [安装没有 CAPI Operator 的 Rancher Turtles](#不使用-cluster-api-capi-operator-作为-helm-依赖安装-rancher-turtles)。 + +安装 Rancher Turtles 需要 CAPI Operator。你可以选择自己处理此依赖项，还是让 Rancher Turtles Helm chart 替你管理它。[使用 CAPI Operator 作为依赖项安装 Rancher Turtles](#使用-cluster-api-capi-operator-作为-helm-依赖项安装-rancher-turtles) 更简单，但是你的最佳选择取决于你的具体配置。 + +CAPI Operator 允许使用声明式方法处理 CAPI provider 的生命周期，扩展了 `clusterctl` 的能力。如果你想了解更多相关内容，可以参考 [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/)。 + +#### 使用 `Cluster API (CAPI) Operator` 作为 Helm 依赖项安装 Rancher Turtles + +1. 添加包含 `rancher-turtles` chart 的 Helm 仓库作为安装的第一步： + +```bash +helm repo add turtles https://rancher.github.io/turtles +helm repo update +``` + +2. 如前面所述，安装 Rancher Turtles 需要 [CAPI Operator](https://github.com/kubernetes-sigs/cluster-api-operator)。Helm chart 可以使用一组最少的参数自动安装： + +```bash +helm install rancher-turtles turtles/rancher-turtles --version \ + -n rancher-turtles-system \ + --dependency-update \ + --create-namespace --wait \ + --timeout 180s +``` + +3. 此操作可能需要几分钟时间，完成后，你可以查看下面列出的已安装的控制器： + +- `rancher-turtles-controller` +- `capi-operator` + +:::note + +- 如果集群中已经有可用的 `cert-manager`，请在安装时通过以下参数将 Rancher Turtles 的此项依赖禁用掉，来阻止冲突： + `--set cluster-api-operator.cert-manager.enabled=false` +- 有关 Rancher Turtles 的版本列表，请参阅 [Turtles 发布页面](https://github.com/rancher/turtles/releases)。 + +::: + +这是最基本的推荐配置，用于管理在核心提供商的命名空间中创建包含所需 CAPI 功能标志（已启用 `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` 和 `EXP_MACHINE_POOL` ）的 secret。要启用其他 CAPI 功能需要启用上述功能标志。 + +如果你需要覆盖默认的行为并使用现有 secret 或添加自定义环境变量，你可以将 secret 名称通过 Helm 参数传入。在这种情况下，你作为负责管理 secret 创建和内容的用户，需要启用的最少特性包括：`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` 和 `EXP_MACHINE_POOL`。 + +```bash +helm install ... + # Passing secret name and namespace for additional environment variables + --set cluster-api-operator.cluster-api.configSecret.name= +``` + +以下是一个用户管理 secret `cluster-api-operator.cluster-api.configSecret.name=variables` 的示例，其中设置了 `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` 和 `EXP_MACHINE_POOL` 功能以及一个额外的自定义变量： + +```yaml title="secret.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: variables + namespace: rancher-turtles-system +type: Opaque +stringData: + CLUSTER_TOPOLOGY: "true" + EXP_CLUSTER_RESOURCE_SET: "true" + EXP_MACHINE_POOL: "true" + CUSTOM_ENV_VAR: "false" +``` + +:::info + +有关 chart 支持的 values 及其用法的详细信息，请参阅 [Helm chart 选项](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) + +::: + +#### 不使用 `Cluster API (CAPI) Operator` 作为 Helm 依赖安装 Rancher Turtles + +:::note + +请记住，如果使用此安装选项，你必须自行管理 CAPI Operator 的安装。你可以参照 Rancher Turtles 文档中的 [CAPI Operator 指南](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) + +::: + +1. 添加包含 `rancher-turtles` chart 的 Helm 仓库作为安装的第一步： + +```bash +helm repo add turtles https://rancher.github.io/turtles +helm repo update +``` + +2. 将 chart 安装到 `rancher-turtles-system` 命名空间： + +```bash +helm install rancher-turtles turtles/rancher-turtles --version + -n rancher-turtles-system + --set cluster-api-operator.enabled=false + --set cluster-api-operator.cluster-api.enabled=false + --create-namespace --wait + --dependency-update +``` + +前面的命令告诉 Helm 忽略将 `cluster-api-operator` 作为依赖项安装。 + +3. 此操作可能需要几分钟，完成后你可以查看下面列出的已安装的控制器： + +- `rancher-turtles-controller` + +## 卸载 Rancher Turtles + +:::caution + +在 Rancher 环境中安装 Rancher Turtles 时，Rancher Turtles 默认会启用 CAPI Operator 清理。这包括清理 CAPI Operator 特定的 webhook 和 deployments，否则会导致 Rancher 配置出现问题。 + +为了简化 Rancher Turtles 卸载（通过 Rancher 或 Helm 命令），官方的 Rancher Turtles Helm chart 包含了一个删除以下内容的 `post-delete` hook： + +- 删除不再需要的 `mutating-webhook-configuration` 和 `validating-webhook-configuration` webhook。 +- 删除不再需要的 CAPI `deployments`。 + +::: + +卸载 Rancher Turtles： + +```bash +helm uninstall -n rancher-turtles-system rancher-turtles --cascade foreground --wait +``` + +这可能需要几分钟才能完成。 + +:::note + +请记住，如果你在安装时使用了不同的名称或不同的命名空间，你需要针对你的特定配置自定义卸载命令。 + +::: + +卸载 Rancher Turtles 后， Rancher 的 `embedded-cluster-api` 功能必须重新启用。 + +1. 创建一个 `feature.yaml` 文件，将 `embedded-cluster-api` 设置为 true： + +```yaml title="feature.yaml" +apiVersion: management.cattle.io/v3 +kind: Feature +metadata: + name: embedded-cluster-api +spec: + value: true +``` + +2. 使用 `kubectl` 将 `feature.yaml` 文件应用到集群： + +```bash +kubectl apply -f feature.yaml +``` + +## 安全 + +[SLSA](https://slsa.dev/spec/v1.0/about) 是一套由行业共识制定的可逐步采用的供应链安全指南。SLSA 制定的规范对软件生产者和消费者都很有用：生产者可以遵循 SLSA 的指导方针，使他们的软件供应链更加安全，消费者可以使用 SLSA 来决定是否信任软件包。 + +Rancher Turtles 满足 [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) 对适当的构建平台、一致的构建过程和来源分布的要求。更多信息请参阅 [Rancher Turtles 安全](https://turtles.docs.rancher.com/docs/security/slsa)文档。 From 489b54c333245a87ba6f1f8a10b52636862e6458 Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Mon, 15 Apr 2024 11:28:59 -0400 Subject: [PATCH 47/67] #593 add password requirements (#1208) * 593 Add password requirments * re-organize page to use tabs, remove redundant material * eng: Bootstrap password has no validation/length requirements, subsequent admin passwords must be 12 chars or longer * update quickstart guides with link to password requirments * link to content in setting up bootstrap password * Apply suggestions from code review Co-authored-by: Lucas Saintarbor * suggestions from code review * update password requirements * helm cli * fix link * another link * Apply suggestions from code review Co-authored-by: Billy Tat * sync 2.8 * syncing 2.7 * adding in helm-cli sync for 2.7 & 2.8 --------- Co-authored-by: Lucas Saintarbor Co-authored-by: Billy Tat --- .../helm-chart-options.md | 16 +---- .../resources/bootstrap-password.md | 61 +++++++++++++++---- .../deploy-rancher-manager/aws.md | 2 +- .../deploy-rancher-manager/azure.md | 2 +- .../deploy-rancher-manager/digitalocean.md | 2 +- .../deploy-rancher-manager/gcp.md | 2 +- .../deploy-rancher-manager/helm-cli.md | 2 +- .../deploy-rancher-manager/hetzner-cloud.md | 2 +- .../deploy-rancher-manager/linode.md | 2 +- .../deploy-rancher-manager/outscale-qs.md | 2 +- .../helm-chart-options.md | 16 +---- .../resources/bootstrap-password.md | 61 +++++++++++++++---- .../deploy-rancher-manager/aws.md | 4 +- .../deploy-rancher-manager/azure.md | 4 +- .../deploy-rancher-manager/digitalocean.md | 4 +- .../deploy-rancher-manager/gcp.md | 4 +- .../deploy-rancher-manager/helm-cli.md | 2 +- .../deploy-rancher-manager/hetzner-cloud.md | 4 +- .../deploy-rancher-manager/linode.md | 4 +- .../deploy-rancher-manager/outscale-qs.md | 4 +- .../helm-chart-options.md | 16 +---- .../resources/bootstrap-password.md | 61 +++++++++++++++---- .../deploy-rancher-manager/aws.md | 2 +- .../deploy-rancher-manager/azure.md | 3 +- .../deploy-rancher-manager/digitalocean.md | 2 +- .../deploy-rancher-manager/gcp.md | 2 +- .../deploy-rancher-manager/helm-cli.md | 2 +- .../deploy-rancher-manager/hetzner-cloud.md | 2 +- .../deploy-rancher-manager/linode.md | 2 +- .../deploy-rancher-manager/outscale-qs.md | 2 +- .../helm-chart-options.md | 16 +---- .../resources/bootstrap-password.md | 61 +++++++++++++++---- .../deploy-rancher-manager/azure.md | 2 +- .../deploy-rancher-manager/digitalocean.md | 4 +- .../deploy-rancher-manager/gcp.md | 3 +- .../deploy-rancher-manager/helm-cli.md | 2 +- .../deploy-rancher-manager/hetzner-cloud.md | 4 +- .../deploy-rancher-manager/linode.md | 2 +- .../deploy-rancher-manager/outscale-qs.md | 2 +- 39 files changed, 249 insertions(+), 141 deletions(-) diff --git a/docs/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/docs/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index 21bc34464e5..19afe0bbb3d 100644 --- a/docs/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/docs/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -17,7 +17,7 @@ For information on enabling experimental features, refer to [this page.](../../. | Option | Default Value | Description | | ------------------------- | ------------- | ---------------------------------------------------------------------------------- | -| `bootstrapPassword` | " " | `string` - Set the [bootstrap password](#bootstrap-password) for the first admin user. After logging in, the admin will need to reset their password. A randomly generated bootstrap password is used if this value is not set. +| `bootstrapPassword` | " " | `string` - Set the [bootstrap password](#bootstrap-password) for the first admin user. After logging in, the admin should reset their password. A randomly generated bootstrap password is used if this value is not set. | `hostname` | " " | `string` - the Fully Qualified Domain Name for your Rancher Server | | `ingress.tls.source` | "rancher" | `string` - Where to get the cert for the ingress. - "rancher, letsEncrypt, secret" | | `letsEncrypt.email` | " " | `string` - Your email address | @@ -67,19 +67,9 @@ For information on enabling experimental features, refer to [this page.](../../. ### Bootstrap Password -When Rancher starts for the first time, a password is randomly generated for the first admin user. When the admin first logs in to Rancher, the UI shows commands that can be used to retrieve the bootstrap password. The admin needs to run those commands and log in with the bootstrap password. Then Rancher gives the admin an opportunity to reset the password. +You can [set a specific bootstrap password](../resources/bootstrap-password.md) during Rancher installation. If you don't set a specific bootstrap password, Rancher randomly generates a password for the first admin account. -If you want to use a specific bootstrap password instead of a randomly generated one, provide the password. - -```plain ---set bootstrapPassword="rancher" -``` - -The password, whether provided or generated, will be stored in a Kubernetes secret. After Rancher is installed, the UI will show instructions for how to retrieve the password using kubectl: - -``` -kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}' -``` +When you log in for the first time, use the bootstrap password you set to log in. If you did not set a bootstrap password, the Rancher UI shows commands that can be used to [retrieve the bootstrap password](../resources/bootstrap-password.md#retrieving-the-bootstrap-password). Run those commands and log in to the account. After you log in for the first time, you are asked to reset the admin password. ### API Audit Log diff --git a/docs/getting-started/installation-and-upgrade/resources/bootstrap-password.md b/docs/getting-started/installation-and-upgrade/resources/bootstrap-password.md index d321c3a99c8..c1ec2ecdcfb 100644 --- a/docs/getting-started/installation-and-upgrade/resources/bootstrap-password.md +++ b/docs/getting-started/installation-and-upgrade/resources/bootstrap-password.md @@ -6,26 +6,63 @@ title: Setting up the Bootstrap Password -When Rancher starts for the first time, a password is randomly generated for the first admin user. When the admin first logs in to Rancher, the UI shows commands that can be used to retrieve the bootstrap password. The admin needs to run those commands and log in with the bootstrap password. Then Rancher gives the admin an opportunity to reset the password. +When you install Rancher, you can set a bootstrap password for the first admin account. -The bootstrap password is randomly generated if it is not set during installation with a variable. For details on how to set the bootstrap password using a variable, see below. +If you choose not to set a bootstrap password, Rancher randomly generates a bootstrap password for the first admin account. -### Specifying the Bootstrap Password in Helm Installs +For details on how to set the bootstrap password, see below. -For a Helm install, users can specify the bootstrap password variable by configuring it in the Helm chart values with `.Values.bootstrapPassword`. +## Password Requirements -The password will be stored in a Kubernetes secret. After Rancher is installed, the UI will show instructions for how to retrieve the password using kubectl: +The bootstrap password can be any length. +When you reset the first admin account's password after first login, the new password must be at least 12 characters long. + +You can [customize the minimum password length](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md#minimum-password-length) for user accounts, within limitations. + +Minimum password length can be any positive integer value between 2 and 256. Decimal values and leading zeroes are not allowed. + +## Specifying the Bootstrap Password + + + + +During [Rancher installation](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md), set `bootstrapPassword` alongside any other flags for the Rancher Helm chart. For example: + +```bash +helm install rancher rancher-/rancher \ + --set bootstrapPassword= ``` + + + + +Pass the following value to the [Docker install command](../other-installation-methods/air-gapped-helm-cli-install/docker-install-commands.md): + +```bash +-e CATTLE_BOOTSTRAP_PASSWORD= +``` + + + + +## Retrieving the Bootstrap Password + +The bootstrap password is stored in the Docker container logs. After Rancher is installed, the UI shows instructions for how to retrieve the password based on your installation method. + + + + +```bash kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}' ``` -### Specifying the Bootstrap Password in Docker Installs - -For a Docker install, you can specify the bootstrap password by passing `-e CATTLE_BOOTSTRAP_PASSWORD=password` to the Docker install command. - -The password will be stored in the Docker container logs. After Rancher is installed, the UI will show instructions for how to retrieve the password using the Docker container ID: + + +```bash +docker logs container-id 2>&1 | grep "Bootstrap Password:" ``` -docker logs container-id 2>&1 | grep "Bootstrap Password:" -``` \ No newline at end of file + + + \ No newline at end of file diff --git a/docs/getting-started/quick-start-guides/deploy-rancher-manager/aws.md b/docs/getting-started/quick-start-guides/deploy-rancher-manager/aws.md index 91b82597680..00712599144 100644 --- a/docs/getting-started/quick-start-guides/deploy-rancher-manager/aws.md +++ b/docs/getting-started/quick-start-guides/deploy-rancher-manager/aws.md @@ -57,7 +57,7 @@ The AWS module just creates an EC2 KeyPair, an EC2 SecurityGroup and an EC2 inst - `aws_access_key` - Amazon AWS Access Key - `aws_secret_key` - Amazon AWS Secret Key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [AWS Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/aws) for more information. Suggestions include: diff --git a/docs/getting-started/quick-start-guides/deploy-rancher-manager/azure.md b/docs/getting-started/quick-start-guides/deploy-rancher-manager/azure.md index 82917ee7857..1cffb5be730 100644 --- a/docs/getting-started/quick-start-guides/deploy-rancher-manager/azure.md +++ b/docs/getting-started/quick-start-guides/deploy-rancher-manager/azure.md @@ -43,7 +43,7 @@ Deploying to Microsoft Azure will incur charges. - `azure_client_id` - Microsoft Azure Client ID - `azure_client_secret` - Microsoft Azure Client Secret - `azure_tenant_id` - Microsoft Azure Tenant ID - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Azure Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/azure) for more information. Suggestions include: diff --git a/docs/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md b/docs/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md index 5d3be8eeda1..40ead75f9b4 100644 --- a/docs/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md +++ b/docs/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md @@ -38,7 +38,7 @@ Deploying to DigitalOcean will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `do_token` - DigitalOcean access key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [DO Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/do) for more information. Suggestions include: diff --git a/docs/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md b/docs/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md index ef465375c60..9dbad648dae 100644 --- a/docs/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md +++ b/docs/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md @@ -39,7 +39,7 @@ Deploying to Google GCP will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `gcp_account_json` - GCP service account file path and file name - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [GCP Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/gcp) for more information. diff --git a/docs/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md b/docs/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md index 07afc518a5a..3e80f451cd9 100644 --- a/docs/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md +++ b/docs/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md @@ -130,7 +130,7 @@ To install a specific Rancher version, use the `--version` flag (e.g., `--versio For Kubernetes v1.25 or later, set `global.cattle.psp.enabled` to `false` when using Rancher v2.7.2-v2.7.4. This is not necessary for Rancher v2.7.5 and above, but you can still manually set the option if you choose. -Note the password requires a minimum of 12 characters. +See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. ``` helm install rancher rancher-latest/rancher \ diff --git a/docs/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md b/docs/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md index eb56bfbe452..3f955ab3153 100644 --- a/docs/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md +++ b/docs/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md @@ -38,7 +38,7 @@ Deploying to Hetzner Cloud will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `hcloud_token` - Hetzner API access key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Hetzner Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/hcloud) for more information. diff --git a/docs/getting-started/quick-start-guides/deploy-rancher-manager/linode.md b/docs/getting-started/quick-start-guides/deploy-rancher-manager/linode.md index 683a0441dd0..47d3cdcf43d 100644 --- a/docs/getting-started/quick-start-guides/deploy-rancher-manager/linode.md +++ b/docs/getting-started/quick-start-guides/deploy-rancher-manager/linode.md @@ -38,7 +38,7 @@ Deploying to Linode will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `linode_token` - The Linode Personal Access Token mentioned above. - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters). + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Linode Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/linode) for more information. Suggestions include: diff --git a/docs/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md b/docs/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md index 5d4e03fc6b9..b6dc5a71495 100644 --- a/docs/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md +++ b/docs/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md @@ -39,7 +39,7 @@ Deploying to Outscale will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `access_key_id` - Outscale access key - `secret_key_id` - Outscale secret key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Outscale Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/outscale) for more information. diff --git a/versioned_docs/version-2.6/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/versioned_docs/version-2.6/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index db62680bc5a..45b46bbc9ee 100644 --- a/versioned_docs/version-2.6/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/versioned_docs/version-2.6/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -17,7 +17,7 @@ For information on enabling experimental features, refer to [this page.](../../. | Option | Default Value | Description | | ------------------------- | ------------- | ---------------------------------------------------------------------------------- | -| `bootstrapPassword` | " " | `string` - Set the [bootstrap password](#bootstrap-password) for the first admin user. After logging in, the admin will need to reset their password. A randomly generated bootstrap password is used if this value is not set. +| `bootstrapPassword` | " " | `string` - Set the [bootstrap password](#bootstrap-password) for the first admin user. After logging in, the admin should reset their password. A randomly generated bootstrap password is used if this value is not set. | `hostname` | " " | `string` - the Fully Qualified Domain Name for your Rancher Server | | `ingress.tls.source` | "rancher" | `string` - Where to get the cert for the ingress. - "rancher, letsEncrypt, secret" | | `letsEncrypt.email` | " " | `string` - Your email address | @@ -64,19 +64,9 @@ For information on enabling experimental features, refer to [this page.](../../. ### Bootstrap Password -When Rancher starts for the first time, a password is randomly generated for the first admin user. When the admin first logs in to Rancher, the UI shows commands that can be used to retrieve the bootstrap password. The admin needs to run those commands and log in with the bootstrap password. Then Rancher gives the admin an opportunity to reset the password. +You can [set a specific bootstrap password](../resources/bootstrap-password.md) during Rancher installation. If you don't set a specific bootstrap password, Rancher randomly generates a password for the first admin account. -If you want to use a specific bootstrap password instead of a randomly generated one, provide the password. - -```plain ---set bootstrapPassword="rancher" -``` - -The password, whether provided or generated, will be stored in a Kubernetes secret. After Rancher is installed, the UI will show instructions for how to retrieve the password using kubectl: - -``` -kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}' -``` +When you log in for the first time, use the bootstrap password you set to log in. If you did not set a bootstrap password, the Rancher UI shows commands that can be used to [retrieve the bootstrap password](../resources/bootstrap-password.md#retrieving-the-bootstrap-password). Run those commands and log in to the account. After you log in for the first time, you are asked to reset the admin password. ### API Audit Log diff --git a/versioned_docs/version-2.6/getting-started/installation-and-upgrade/resources/bootstrap-password.md b/versioned_docs/version-2.6/getting-started/installation-and-upgrade/resources/bootstrap-password.md index d321c3a99c8..2c52b4501b3 100644 --- a/versioned_docs/version-2.6/getting-started/installation-and-upgrade/resources/bootstrap-password.md +++ b/versioned_docs/version-2.6/getting-started/installation-and-upgrade/resources/bootstrap-password.md @@ -6,26 +6,63 @@ title: Setting up the Bootstrap Password -When Rancher starts for the first time, a password is randomly generated for the first admin user. When the admin first logs in to Rancher, the UI shows commands that can be used to retrieve the bootstrap password. The admin needs to run those commands and log in with the bootstrap password. Then Rancher gives the admin an opportunity to reset the password. +When you install Rancher, you can set a bootstrap password for the first admin account. -The bootstrap password is randomly generated if it is not set during installation with a variable. For details on how to set the bootstrap password using a variable, see below. +If you choose not to set a bootstrap password, Rancher randomly generates a bootstrap password for the first admin account. -### Specifying the Bootstrap Password in Helm Installs +For details on how to set the bootstrap password, see below. -For a Helm install, users can specify the bootstrap password variable by configuring it in the Helm chart values with `.Values.bootstrapPassword`. +## Password Requirements -The password will be stored in a Kubernetes secret. After Rancher is installed, the UI will show instructions for how to retrieve the password using kubectl: +The bootstrap password can be any length. +When you reset the first admin account's password after first login, the new password must be at least 12 characters long. + +In Rancher v2.6.9 and later, you can [customize the minimum password length](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md#minimum-password-length) for user accounts, within limitations. + +Minimum password length can be any positive integer value between 2 and 256. Decimal values and leading zeroes are not allowed. + +## Specifying the Bootstrap Password + + + + +During [Rancher installation](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md), set `bootstrapPassword` alongside any other flags for the Rancher Helm chart. For example: + +```bash +helm install rancher rancher-/rancher \ + --set bootstrapPassword= ``` + + + + +Pass the following value to the [Docker install command](../other-installation-methods/air-gapped-helm-cli-install/docker-install-commands.md): + +```bash +-e CATTLE_BOOTSTRAP_PASSWORD= +``` + + + + +## Retrieving the Bootstrap Password + +The bootstrap password is stored in the Docker container logs. After Rancher is installed, the UI shows instructions for how to retrieve the password based on your installation method. + + + + +```bash kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}' ``` -### Specifying the Bootstrap Password in Docker Installs - -For a Docker install, you can specify the bootstrap password by passing `-e CATTLE_BOOTSTRAP_PASSWORD=password` to the Docker install command. - -The password will be stored in the Docker container logs. After Rancher is installed, the UI will show instructions for how to retrieve the password using the Docker container ID: + + +```bash +docker logs container-id 2>&1 | grep "Bootstrap Password:" ``` -docker logs container-id 2>&1 | grep "Bootstrap Password:" -``` \ No newline at end of file + + + diff --git a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/aws.md b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/aws.md index 91b82597680..ac48145d491 100644 --- a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/aws.md +++ b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/aws.md @@ -57,8 +57,8 @@ The AWS module just creates an EC2 KeyPair, an EC2 SecurityGroup and an EC2 inst - `aws_access_key` - Amazon AWS Access Key - `aws_secret_key` - Amazon AWS Secret Key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) - + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. + 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [AWS Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/aws) for more information. Suggestions include: diff --git a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/azure.md b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/azure.md index 82917ee7857..f69d835c171 100644 --- a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/azure.md +++ b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/azure.md @@ -43,8 +43,8 @@ Deploying to Microsoft Azure will incur charges. - `azure_client_id` - Microsoft Azure Client ID - `azure_client_secret` - Microsoft Azure Client Secret - `azure_tenant_id` - Microsoft Azure Tenant ID - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) - + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. + 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Azure Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/azure) for more information. Suggestions include: - `azure_location` - Microsoft Azure region, choose the closest instead of the default (`East US`) diff --git a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md index 5d3be8eeda1..8a041542a22 100644 --- a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md +++ b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md @@ -38,8 +38,8 @@ Deploying to DigitalOcean will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `do_token` - DigitalOcean access key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) - + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. + 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [DO Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/do) for more information. Suggestions include: - `do_region` - DigitalOcean region, choose the closest instead of the default (`nyc1`) diff --git a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md index ef465375c60..4b562ad84c9 100644 --- a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md +++ b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md @@ -39,8 +39,8 @@ Deploying to Google GCP will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `gcp_account_json` - GCP service account file path and file name - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) - + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. + 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [GCP Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/gcp) for more information. Suggestions include: diff --git a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md index 2232371a091..0c8b8ff8047 100644 --- a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md +++ b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md @@ -129,7 +129,7 @@ The final command to install Rancher is below. The command requires a domain nam To install a specific Rancher version, use the `--version` flag (e.g., `--version 2.6.6`). Otherwise, the latest Rancher is installed by default. Refer to [Choosing a Rancher Version](../../installation-and-upgrade/resources/choose-a-rancher-version.md). -Note the password requires a minimum of 12 characters. +See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. ``` helm install rancher rancher-latest/rancher \ diff --git a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md index eb56bfbe452..ea969ecb4db 100644 --- a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md +++ b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md @@ -38,8 +38,8 @@ Deploying to Hetzner Cloud will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `hcloud_token` - Hetzner API access key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) - + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. + 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Hetzner Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/hcloud) for more information. Suggestions include: diff --git a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/linode.md b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/linode.md index 654ff7195cd..0153d15ba0f 100644 --- a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/linode.md +++ b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/linode.md @@ -38,8 +38,8 @@ Deploying to Linode will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `linode_token` - The Linode Personal Access Token mentioned above. - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters). - + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. + 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Linode Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/linode) for more information. Suggestions include: - `linode_region` - The target Linode region to provision the server and cluster in. diff --git a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md index 5d4e03fc6b9..a5debfcb9da 100644 --- a/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md +++ b/versioned_docs/version-2.6/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md @@ -39,8 +39,8 @@ Deploying to Outscale will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `access_key_id` - Outscale access key - `secret_key_id` - Outscale secret key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) - + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. + 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Outscale Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/outscale) for more information. Suggestions include: diff --git a/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index 98cec0640f5..5e4d9dfc7d9 100644 --- a/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -17,7 +17,7 @@ For information on enabling experimental features, refer to [this page.](../../. | Option | Default Value | Description | | ------------------------- | ------------- | ---------------------------------------------------------------------------------- | -| `bootstrapPassword` | " " | `string` - Set the [bootstrap password](#bootstrap-password) for the first admin user. After logging in, the admin will need to reset their password. A randomly generated bootstrap password is used if this value is not set. +| `bootstrapPassword` | " " | `string` - Set the [bootstrap password](#bootstrap-password) for the first admin user. After logging in, the admin should reset their password. A randomly generated bootstrap password is used if this value is not set. | `hostname` | " " | `string` - the Fully Qualified Domain Name for your Rancher Server | | `ingress.tls.source` | "rancher" | `string` - Where to get the cert for the ingress. - "rancher, letsEncrypt, secret" | | `letsEncrypt.email` | " " | `string` - Your email address | @@ -67,19 +67,9 @@ For information on enabling experimental features, refer to [this page.](../../. ### Bootstrap Password -When Rancher starts for the first time, a password is randomly generated for the first admin user. When the admin first logs in to Rancher, the UI shows commands that can be used to retrieve the bootstrap password. The admin needs to run those commands and log in with the bootstrap password. Then Rancher gives the admin an opportunity to reset the password. +You can [set a specific bootstrap password](../resources/bootstrap-password.md) during Rancher installation. If you don't set a specific bootstrap password, Rancher randomly generates a password for the first admin account. -If you want to use a specific bootstrap password instead of a randomly generated one, provide the password. - -```plain ---set bootstrapPassword="rancher" -``` - -The password, whether provided or generated, will be stored in a Kubernetes secret. After Rancher is installed, the UI will show instructions for how to retrieve the password using kubectl: - -``` -kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}' -``` +When you log in for the first time, use the bootstrap password you set to log in. If you did not set a bootstrap password, the Rancher UI shows commands that can be used to [retrieve the bootstrap password](../resources/bootstrap-password.md#retrieving-the-bootstrap-password). Run those commands and log in to the account. After you log in for the first time, you are asked to reset the admin password. ### API Audit Log diff --git a/versioned_docs/version-2.7/getting-started/installation-and-upgrade/resources/bootstrap-password.md b/versioned_docs/version-2.7/getting-started/installation-and-upgrade/resources/bootstrap-password.md index d321c3a99c8..864082d630a 100644 --- a/versioned_docs/version-2.7/getting-started/installation-and-upgrade/resources/bootstrap-password.md +++ b/versioned_docs/version-2.7/getting-started/installation-and-upgrade/resources/bootstrap-password.md @@ -6,26 +6,63 @@ title: Setting up the Bootstrap Password -When Rancher starts for the first time, a password is randomly generated for the first admin user. When the admin first logs in to Rancher, the UI shows commands that can be used to retrieve the bootstrap password. The admin needs to run those commands and log in with the bootstrap password. Then Rancher gives the admin an opportunity to reset the password. +When you install Rancher, you can set a bootstrap password for the first admin account. -The bootstrap password is randomly generated if it is not set during installation with a variable. For details on how to set the bootstrap password using a variable, see below. +If you choose not to set a bootstrap password, Rancher randomly generates a bootstrap password for the first admin account. -### Specifying the Bootstrap Password in Helm Installs +For details on how to set the bootstrap password, see below. -For a Helm install, users can specify the bootstrap password variable by configuring it in the Helm chart values with `.Values.bootstrapPassword`. +## Password Requirements -The password will be stored in a Kubernetes secret. After Rancher is installed, the UI will show instructions for how to retrieve the password using kubectl: +The bootstrap password can be any length. +When you reset the first admin account's password after first login, the new password must be at least 12 characters long. + +You can [customize the minimum password length](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md#minimum-password-length) for user accounts, within limitations. + +Minimum password length can be any positive integer value between 2 and 256. Decimal values and leading zeroes are not allowed. + +## Specifying the Bootstrap Password + + + + +During [Rancher installation](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md), set `bootstrapPassword` alongside any other flags for the Rancher Helm chart. For example: + +```bash +helm install rancher rancher-/rancher \ + --set bootstrapPassword= ``` + + + + +Pass the following value to the [Docker install command](../other-installation-methods/air-gapped-helm-cli-install/docker-install-commands.md): + +```bash +-e CATTLE_BOOTSTRAP_PASSWORD= +``` + + + + +## Retrieving the Bootstrap Password + +The bootstrap password is stored in the Docker container logs. After Rancher is installed, the UI shows instructions for how to retrieve the password based on your installation method. + + + + +```bash kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}' ``` -### Specifying the Bootstrap Password in Docker Installs - -For a Docker install, you can specify the bootstrap password by passing `-e CATTLE_BOOTSTRAP_PASSWORD=password` to the Docker install command. - -The password will be stored in the Docker container logs. After Rancher is installed, the UI will show instructions for how to retrieve the password using the Docker container ID: + + +```bash +docker logs container-id 2>&1 | grep "Bootstrap Password:" ``` -docker logs container-id 2>&1 | grep "Bootstrap Password:" -``` \ No newline at end of file + + + diff --git a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/aws.md b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/aws.md index 91b82597680..00712599144 100644 --- a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/aws.md +++ b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/aws.md @@ -57,7 +57,7 @@ The AWS module just creates an EC2 KeyPair, an EC2 SecurityGroup and an EC2 inst - `aws_access_key` - Amazon AWS Access Key - `aws_secret_key` - Amazon AWS Secret Key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [AWS Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/aws) for more information. Suggestions include: diff --git a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/azure.md b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/azure.md index 82917ee7857..a3585d3cb7a 100644 --- a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/azure.md +++ b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/azure.md @@ -43,8 +43,7 @@ Deploying to Microsoft Azure will incur charges. - `azure_client_id` - Microsoft Azure Client ID - `azure_client_secret` - Microsoft Azure Client Secret - `azure_tenant_id` - Microsoft Azure Tenant ID - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) - + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Azure Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/azure) for more information. Suggestions include: - `azure_location` - Microsoft Azure region, choose the closest instead of the default (`East US`) diff --git a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md index 5d3be8eeda1..40ead75f9b4 100644 --- a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md +++ b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md @@ -38,7 +38,7 @@ Deploying to DigitalOcean will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `do_token` - DigitalOcean access key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [DO Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/do) for more information. Suggestions include: diff --git a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md index ef465375c60..9dbad648dae 100644 --- a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md +++ b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md @@ -39,7 +39,7 @@ Deploying to Google GCP will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `gcp_account_json` - GCP service account file path and file name - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [GCP Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/gcp) for more information. diff --git a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md index 07afc518a5a..3e80f451cd9 100644 --- a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md +++ b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md @@ -130,7 +130,7 @@ To install a specific Rancher version, use the `--version` flag (e.g., `--versio For Kubernetes v1.25 or later, set `global.cattle.psp.enabled` to `false` when using Rancher v2.7.2-v2.7.4. This is not necessary for Rancher v2.7.5 and above, but you can still manually set the option if you choose. -Note the password requires a minimum of 12 characters. +See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. ``` helm install rancher rancher-latest/rancher \ diff --git a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md index eb56bfbe452..3f955ab3153 100644 --- a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md +++ b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md @@ -38,7 +38,7 @@ Deploying to Hetzner Cloud will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `hcloud_token` - Hetzner API access key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Hetzner Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/hcloud) for more information. diff --git a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/linode.md b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/linode.md index 654ff7195cd..56478d7b947 100644 --- a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/linode.md +++ b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/linode.md @@ -38,7 +38,7 @@ Deploying to Linode will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `linode_token` - The Linode Personal Access Token mentioned above. - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters). + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Linode Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/linode) for more information. Suggestions include: diff --git a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md index 5d4e03fc6b9..48c8f93f8d1 100644 --- a/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md +++ b/versioned_docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md @@ -39,7 +39,7 @@ Deploying to Outscale will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `access_key_id` - Outscale access key - `secret_key_id` - Outscale secret key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) +- `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Outscale Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/outscale) for more information. diff --git a/versioned_docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/versioned_docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index 8b614f2da70..4f125600265 100644 --- a/versioned_docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/versioned_docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -17,7 +17,7 @@ For information on enabling experimental features, refer to [this page.](../../. | Option | Default Value | Description | | ------------------------- | ------------- | ---------------------------------------------------------------------------------- | -| `bootstrapPassword` | " " | `string` - Set the [bootstrap password](#bootstrap-password) for the first admin user. After logging in, the admin will need to reset their password. A randomly generated bootstrap password is used if this value is not set. +| `bootstrapPassword` | " " | `string` - Set the [bootstrap password](#bootstrap-password) for the first admin user. After logging in, the admin should reset their password. A randomly generated bootstrap password is used if this value is not set. | `hostname` | " " | `string` - the Fully Qualified Domain Name for your Rancher Server | | `ingress.tls.source` | "rancher" | `string` - Where to get the cert for the ingress. - "rancher, letsEncrypt, secret" | | `letsEncrypt.email` | " " | `string` - Your email address | @@ -67,19 +67,9 @@ For information on enabling experimental features, refer to [this page.](../../. ### Bootstrap Password -When Rancher starts for the first time, a password is randomly generated for the first admin user. When the admin first logs in to Rancher, the UI shows commands that can be used to retrieve the bootstrap password. The admin needs to run those commands and log in with the bootstrap password. Then Rancher gives the admin an opportunity to reset the password. +You can [set a specific bootstrap password](../resources/bootstrap-password.md) during Rancher installation. If you don't set a specific bootstrap password, Rancher randomly generates a password for the first admin account. -If you want to use a specific bootstrap password instead of a randomly generated one, provide the password. - -```plain ---set bootstrapPassword="rancher" -``` - -The password, whether provided or generated, will be stored in a Kubernetes secret. After Rancher is installed, the UI will show instructions for how to retrieve the password using kubectl: - -``` -kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}' -``` +When you log in for the first time, use the bootstrap password you set to log in. If you did not set a bootstrap password, the Rancher UI shows commands that can be used to [retrieve the bootstrap password](../resources/bootstrap-password.md#retrieving-the-bootstrap-password). Run those commands and log in to the account. After you log in for the first time, you are asked to reset the admin password. ### API Audit Log diff --git a/versioned_docs/version-2.8/getting-started/installation-and-upgrade/resources/bootstrap-password.md b/versioned_docs/version-2.8/getting-started/installation-and-upgrade/resources/bootstrap-password.md index d321c3a99c8..c1ec2ecdcfb 100644 --- a/versioned_docs/version-2.8/getting-started/installation-and-upgrade/resources/bootstrap-password.md +++ b/versioned_docs/version-2.8/getting-started/installation-and-upgrade/resources/bootstrap-password.md @@ -6,26 +6,63 @@ title: Setting up the Bootstrap Password -When Rancher starts for the first time, a password is randomly generated for the first admin user. When the admin first logs in to Rancher, the UI shows commands that can be used to retrieve the bootstrap password. The admin needs to run those commands and log in with the bootstrap password. Then Rancher gives the admin an opportunity to reset the password. +When you install Rancher, you can set a bootstrap password for the first admin account. -The bootstrap password is randomly generated if it is not set during installation with a variable. For details on how to set the bootstrap password using a variable, see below. +If you choose not to set a bootstrap password, Rancher randomly generates a bootstrap password for the first admin account. -### Specifying the Bootstrap Password in Helm Installs +For details on how to set the bootstrap password, see below. -For a Helm install, users can specify the bootstrap password variable by configuring it in the Helm chart values with `.Values.bootstrapPassword`. +## Password Requirements -The password will be stored in a Kubernetes secret. After Rancher is installed, the UI will show instructions for how to retrieve the password using kubectl: +The bootstrap password can be any length. +When you reset the first admin account's password after first login, the new password must be at least 12 characters long. + +You can [customize the minimum password length](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/manage-users-and-groups.md#minimum-password-length) for user accounts, within limitations. + +Minimum password length can be any positive integer value between 2 and 256. Decimal values and leading zeroes are not allowed. + +## Specifying the Bootstrap Password + + + + +During [Rancher installation](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md), set `bootstrapPassword` alongside any other flags for the Rancher Helm chart. For example: + +```bash +helm install rancher rancher-/rancher \ + --set bootstrapPassword= ``` + + + + +Pass the following value to the [Docker install command](../other-installation-methods/air-gapped-helm-cli-install/docker-install-commands.md): + +```bash +-e CATTLE_BOOTSTRAP_PASSWORD= +``` + + + + +## Retrieving the Bootstrap Password + +The bootstrap password is stored in the Docker container logs. After Rancher is installed, the UI shows instructions for how to retrieve the password based on your installation method. + + + + +```bash kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ .data.bootstrapPassword|base64decode}}{{ "\n" }}' ``` -### Specifying the Bootstrap Password in Docker Installs - -For a Docker install, you can specify the bootstrap password by passing `-e CATTLE_BOOTSTRAP_PASSWORD=password` to the Docker install command. - -The password will be stored in the Docker container logs. After Rancher is installed, the UI will show instructions for how to retrieve the password using the Docker container ID: + + +```bash +docker logs container-id 2>&1 | grep "Bootstrap Password:" ``` -docker logs container-id 2>&1 | grep "Bootstrap Password:" -``` \ No newline at end of file + + + \ No newline at end of file diff --git a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/azure.md b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/azure.md index 82917ee7857..1cffb5be730 100644 --- a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/azure.md +++ b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/azure.md @@ -43,7 +43,7 @@ Deploying to Microsoft Azure will incur charges. - `azure_client_id` - Microsoft Azure Client ID - `azure_client_secret` - Microsoft Azure Client Secret - `azure_tenant_id` - Microsoft Azure Tenant ID - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Azure Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/azure) for more information. Suggestions include: diff --git a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md index 5d3be8eeda1..8a041542a22 100644 --- a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md +++ b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md @@ -38,8 +38,8 @@ Deploying to DigitalOcean will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `do_token` - DigitalOcean access key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) - + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. + 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [DO Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/do) for more information. Suggestions include: - `do_region` - DigitalOcean region, choose the closest instead of the default (`nyc1`) diff --git a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md index ef465375c60..c04e6b52353 100644 --- a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md +++ b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md @@ -39,7 +39,8 @@ Deploying to Google GCP will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `gcp_account_json` - GCP service account file path and file name - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. + 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [GCP Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/gcp) for more information. diff --git a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md index 07afc518a5a..3e80f451cd9 100644 --- a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md +++ b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md @@ -130,7 +130,7 @@ To install a specific Rancher version, use the `--version` flag (e.g., `--versio For Kubernetes v1.25 or later, set `global.cattle.psp.enabled` to `false` when using Rancher v2.7.2-v2.7.4. This is not necessary for Rancher v2.7.5 and above, but you can still manually set the option if you choose. -Note the password requires a minimum of 12 characters. +See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. ``` helm install rancher rancher-latest/rancher \ diff --git a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md index eb56bfbe452..ea969ecb4db 100644 --- a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md +++ b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md @@ -38,8 +38,8 @@ Deploying to Hetzner Cloud will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `hcloud_token` - Hetzner API access key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) - + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. + 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Hetzner Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/hcloud) for more information. Suggestions include: diff --git a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/linode.md b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/linode.md index 654ff7195cd..56478d7b947 100644 --- a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/linode.md +++ b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/linode.md @@ -38,7 +38,7 @@ Deploying to Linode will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `linode_token` - The Linode Personal Access Token mentioned above. - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters). + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Linode Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/linode) for more information. Suggestions include: diff --git a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md index 5d4e03fc6b9..b6dc5a71495 100644 --- a/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md +++ b/versioned_docs/version-2.8/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md @@ -39,7 +39,7 @@ Deploying to Outscale will incur charges. 4. Edit `terraform.tfvars` and customize the following variables: - `access_key_id` - Outscale access key - `secret_key_id` - Outscale secret key - - `rancher_server_admin_password` - Admin password for created Rancher server (minimum 12 characters) + - `rancher_server_admin_password` - Admin password for created Rancher server. See [Setting up the Bootstrap Password](../../installation-and-upgrade/resources/bootstrap-password.md#password-requirements) for password requirments. 5. **Optional:** Modify optional variables within `terraform.tfvars`. See the [Quickstart Readme](https://github.com/rancher/quickstart) and the [Outscale Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/outscale) for more information. From cef7e9597042028da217300725e6ff885a22cfe0 Mon Sep 17 00:00:00 2001 From: Mateus Valgueiro Date: Mon, 15 Apr 2024 15:15:35 -0300 Subject: [PATCH 48/67] Fix name of option being used on doc (#1235) --- .../installation-references/helm-chart-options.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/docs/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index 19afe0bbb3d..35ebfaa9082 100644 --- a/docs/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/docs/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -153,7 +153,7 @@ Rancher supports CIDR notation ranges in this list. When not including sensitive data, the `proxy` or `extraEnv` chart options can be used. When using `extraEnv` the `noProxy` Helm option is ignored. Therefore, the `NO_PROXY` environment variable must also be set with `extraEnv`. -The following is an example of setting proxy using the `extraEnv` chart option: +The following is an example of setting proxy using the `proxy` chart option: ```plain --set proxy="http:///" From b1aff707a9c4e4d86f96d906bcc9a1be43d05ab1 Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Mon, 15 Apr 2024 14:16:53 -0400 Subject: [PATCH 49/67] sync versioned-docs with 1235 (#1237) --- .../installation-references/helm-chart-options.md | 2 +- .../installation-references/helm-chart-options.md | 2 +- .../installation-references/helm-chart-options.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/versioned_docs/version-2.6/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/versioned_docs/version-2.6/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index 45b46bbc9ee..911c5dc2477 100644 --- a/versioned_docs/version-2.6/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/versioned_docs/version-2.6/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -150,7 +150,7 @@ Rancher supports CIDR notation ranges in this list. When not including sensitive data, the `proxy` or `extraEnv` chart options can be used. When using `extraEnv` the `noProxy` Helm option is ignored. Therefore, the `NO_PROXY` environment variable must also be set with `extraEnv`. -The following is an example of setting proxy using the `extraEnv` chart option: +The following is an example of setting proxy using the `proxy` chart option: ```plain --set proxy="http:///" diff --git a/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index 5e4d9dfc7d9..c6ff10ea455 100644 --- a/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/versioned_docs/version-2.7/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -153,7 +153,7 @@ Rancher supports CIDR notation ranges in this list. When not including sensitive data, the `proxy` or `extraEnv` chart options can be used. When using `extraEnv` the `noProxy` Helm option is ignored. Therefore, the `NO_PROXY` environment variable must also be set with `extraEnv`. -The following is an example of setting proxy using the `extraEnv` chart option: +The following is an example of setting proxy using the `proxy` chart option: ```plain --set proxy="http:///" diff --git a/versioned_docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/versioned_docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index 4f125600265..5daf4fc4141 100644 --- a/versioned_docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/versioned_docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -153,7 +153,7 @@ Rancher supports CIDR notation ranges in this list. When not including sensitive data, the `proxy` or `extraEnv` chart options can be used. When using `extraEnv` the `noProxy` Helm option is ignored. Therefore, the `NO_PROXY` environment variable must also be set with `extraEnv`. -The following is an example of setting proxy using the `extraEnv` chart option: +The following is an example of setting proxy using the `proxy` chart option: ```plain --set proxy="http:///" From 5507bef49dc7e346c829b4b5a9a4ac585af3b0e0 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 15 Apr 2024 11:51:12 -0700 Subject: [PATCH 50/67] Adding an update to the CAPI Overview page specifying where to disable and remove webhooks and the embedded-cluster-api, as well as adding links to the CAPI site on CAPI provider information. Signed-off-by: Sunil Singh --- .../cluster-api/overview.md | 45 +++++++++++++++---- .../cluster-api/overview.md | 45 +++++++++++++++---- .../cluster-api/overview.md | 45 +++++++++++++++---- 3 files changed, 111 insertions(+), 24 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 7d56e974e75..20e1d0d067c 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -12,6 +12,12 @@ Below is a visual representation of the key components of Rancher Turtles and th ![overview](/img/30000ft_view.png) +## Security + +[SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. + +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/docs/security/slsa) document. + ## Prerequisites Before installing Rancher Turtles in your Rancher environment, you must disable Rancher's `embedded-cluster-api` functionality. This also includes cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. @@ -21,6 +27,35 @@ To simplify setting up Rancher for installing Rancher Turtles, the official Ranc - Disables the `embedded-cluster-api` feature in Rancher. - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. +The above webhooks can be removed through the Rancher UI as well and are accessed from the left-hand navigation from your local cluster under the **More Resources** > **Admission** section or found by using the **Resource Search** field and inputting the webhook names. Additionally, the following `kubectl` commands can also be used to remove the necessary webhooks: + +```console +kubectl delete mutatingwebhookconfiguration.admissionregistration.k8s.io mutating-webhook-configuration +``` + +```console +kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io validating-webhook-configuration +``` + +Use the following example to disable the `embedded-cluster-api` feature from the console: + +1. Create a `feature.yaml` file, with `embedded-cluster-api` set to false: + +```yaml title="feature.yaml" +apiVersion: management.cattle.io/v3 +kind: Feature +metadata: + name: embedded-cluster-api +spec: + value: false +``` + +2. Use `kubectl` to apply the `feature.yaml` file to the cluster: + +```bash +kubectl apply -f feature.yaml +``` + ## Installing the Rancher Turtles Operator You can install the Rancher Turtles operator via the Rancher UI, or with Helm. The first method is recommended for most environments. @@ -63,14 +98,14 @@ This demo illustrates how to use the Rancher UI to install Rancher Turtles, crea ### Installing via Helm -There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: +There are two ways to install Rancher Turtles with Helm, depending on whether you include the [CAPI Operator](https://github.com/kubernetes-sigs/cluster-api-operator) as a dependency: - [Install Rancher Turtles with CAPI Operator as a dependency](#installing-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). - [Install Rancher Turtles without CAPI Operator](#installing-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#installing-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. -The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). +The CAPI Operator allows for handling the lifecycle of [CAPI providers](https://turtles.docs.rancher.com/tasks/capi-operator/installing_core_provider) using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). #### Installing Rancher Turtles with `Cluster API (CAPI) Operator` as a Helm dependency @@ -213,9 +248,3 @@ spec: ```bash kubectl apply -f feature.yaml ``` - -## Security - -[SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. - -Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/docs/security/slsa) document. diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index 7d56e974e75..20e1d0d067c 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -12,6 +12,12 @@ Below is a visual representation of the key components of Rancher Turtles and th ![overview](/img/30000ft_view.png) +## Security + +[SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. + +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/docs/security/slsa) document. + ## Prerequisites Before installing Rancher Turtles in your Rancher environment, you must disable Rancher's `embedded-cluster-api` functionality. This also includes cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. @@ -21,6 +27,35 @@ To simplify setting up Rancher for installing Rancher Turtles, the official Ranc - Disables the `embedded-cluster-api` feature in Rancher. - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. +The above webhooks can be removed through the Rancher UI as well and are accessed from the left-hand navigation from your local cluster under the **More Resources** > **Admission** section or found by using the **Resource Search** field and inputting the webhook names. Additionally, the following `kubectl` commands can also be used to remove the necessary webhooks: + +```console +kubectl delete mutatingwebhookconfiguration.admissionregistration.k8s.io mutating-webhook-configuration +``` + +```console +kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io validating-webhook-configuration +``` + +Use the following example to disable the `embedded-cluster-api` feature from the console: + +1. Create a `feature.yaml` file, with `embedded-cluster-api` set to false: + +```yaml title="feature.yaml" +apiVersion: management.cattle.io/v3 +kind: Feature +metadata: + name: embedded-cluster-api +spec: + value: false +``` + +2. Use `kubectl` to apply the `feature.yaml` file to the cluster: + +```bash +kubectl apply -f feature.yaml +``` + ## Installing the Rancher Turtles Operator You can install the Rancher Turtles operator via the Rancher UI, or with Helm. The first method is recommended for most environments. @@ -63,14 +98,14 @@ This demo illustrates how to use the Rancher UI to install Rancher Turtles, crea ### Installing via Helm -There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: +There are two ways to install Rancher Turtles with Helm, depending on whether you include the [CAPI Operator](https://github.com/kubernetes-sigs/cluster-api-operator) as a dependency: - [Install Rancher Turtles with CAPI Operator as a dependency](#installing-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). - [Install Rancher Turtles without CAPI Operator](#installing-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#installing-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. -The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). +The CAPI Operator allows for handling the lifecycle of [CAPI providers](https://turtles.docs.rancher.com/tasks/capi-operator/installing_core_provider) using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). #### Installing Rancher Turtles with `Cluster API (CAPI) Operator` as a Helm dependency @@ -213,9 +248,3 @@ spec: ```bash kubectl apply -f feature.yaml ``` - -## Security - -[SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. - -Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/docs/security/slsa) document. diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index 7d56e974e75..20e1d0d067c 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -12,6 +12,12 @@ Below is a visual representation of the key components of Rancher Turtles and th ![overview](/img/30000ft_view.png) +## Security + +[SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. + +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/docs/security/slsa) document. + ## Prerequisites Before installing Rancher Turtles in your Rancher environment, you must disable Rancher's `embedded-cluster-api` functionality. This also includes cleaning up Rancher-specific webhooks that otherwise would conflict with CAPI ones. @@ -21,6 +27,35 @@ To simplify setting up Rancher for installing Rancher Turtles, the official Ranc - Disables the `embedded-cluster-api` feature in Rancher. - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. +The above webhooks can be removed through the Rancher UI as well and are accessed from the left-hand navigation from your local cluster under the **More Resources** > **Admission** section or found by using the **Resource Search** field and inputting the webhook names. Additionally, the following `kubectl` commands can also be used to remove the necessary webhooks: + +```console +kubectl delete mutatingwebhookconfiguration.admissionregistration.k8s.io mutating-webhook-configuration +``` + +```console +kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io validating-webhook-configuration +``` + +Use the following example to disable the `embedded-cluster-api` feature from the console: + +1. Create a `feature.yaml` file, with `embedded-cluster-api` set to false: + +```yaml title="feature.yaml" +apiVersion: management.cattle.io/v3 +kind: Feature +metadata: + name: embedded-cluster-api +spec: + value: false +``` + +2. Use `kubectl` to apply the `feature.yaml` file to the cluster: + +```bash +kubectl apply -f feature.yaml +``` + ## Installing the Rancher Turtles Operator You can install the Rancher Turtles operator via the Rancher UI, or with Helm. The first method is recommended for most environments. @@ -63,14 +98,14 @@ This demo illustrates how to use the Rancher UI to install Rancher Turtles, crea ### Installing via Helm -There are two ways to install Rancher Turtles with Helm, depending on whether you include the CAPI operator as a dependency: +There are two ways to install Rancher Turtles with Helm, depending on whether you include the [CAPI Operator](https://github.com/kubernetes-sigs/cluster-api-operator) as a dependency: - [Install Rancher Turtles with CAPI Operator as a dependency](#installing-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency). - [Install Rancher Turtles without CAPI Operator](#installing-rancher-turtles-without-cluster-api-capi-operator-as-a-helm-dependency). The CAPI Operator is required for installing Rancher Turtles. You can choose whether you want to take care of this dependency yourself or let the Rancher Turtles Helm chart manage it for you. [Installing Turtles as a dependency](#installing-rancher-turtles-with-cluster-api-capi-operator-as-a-helm-dependency) is simpler, but your best option depends on your specific configuration. -The CAPI Operator allows for handling the lifecycle of CAPI providers using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). +The CAPI Operator allows for handling the lifecycle of [CAPI providers](https://turtles.docs.rancher.com/tasks/capi-operator/installing_core_provider) using a declarative approach, extending the capabilities of `clusterctl`. If you want to learn more about it, you can refer to [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/). #### Installing Rancher Turtles with `Cluster API (CAPI) Operator` as a Helm dependency @@ -213,9 +248,3 @@ spec: ```bash kubectl apply -f feature.yaml ``` - -## Security - -[SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. - -Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/docs/security/slsa) document. From 7b7d140cf92e9f6f6232fd3213271d86013c93ea Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 15 Apr 2024 15:01:19 -0700 Subject: [PATCH 51/67] Updating the links after review. Signed-off-by: Sunil Singh --- docs/integrations-in-rancher/cluster-api/overview.md | 2 +- .../version-2.7/integrations-in-rancher/cluster-api/overview.md | 2 +- .../version-2.8/integrations-in-rancher/cluster-api/overview.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 20e1d0d067c..4ff36435ca5 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -16,7 +16,7 @@ Below is a visual representation of the key components of Rancher Turtles and th [SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. -Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/docs/security/slsa) document. +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/security/slsa) document. ## Prerequisites diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index 20e1d0d067c..4ff36435ca5 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -16,7 +16,7 @@ Below is a visual representation of the key components of Rancher Turtles and th [SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. -Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/docs/security/slsa) document. +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/security/slsa) document. ## Prerequisites diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index 20e1d0d067c..4ff36435ca5 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -16,7 +16,7 @@ Below is a visual representation of the key components of Rancher Turtles and th [SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. -Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/docs/security/slsa) document. +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/security/slsa) document. ## Prerequisites From 2882b4ad6d2bdf9cb62d26e660daeae63028dfa7 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Mon, 15 Apr 2024 15:41:16 -0700 Subject: [PATCH 52/67] Adjusting the prereq section with clear steps and updating the Security section after review. Signed-off-by: Sunil Singh --- .../cluster-api/overview.md | 16 +++++++++++++--- .../cluster-api/overview.md | 16 +++++++++++++--- .../cluster-api/overview.md | 16 +++++++++++++--- 3 files changed, 39 insertions(+), 9 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 4ff36435ca5..970430093d8 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -14,9 +14,9 @@ Below is a visual representation of the key components of Rancher Turtles and th ## Security -[SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. +As defined by [Supply-chain Levels for Software Artifacts (SLSA)](https://slsa.dev/spec/v1.0/about), SLSA is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. -Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/security/slsa) document. +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements as an appropriate hardened build platform, with consistent build processes, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/security/slsa) document. ## Prerequisites @@ -27,7 +27,17 @@ To simplify setting up Rancher for installing Rancher Turtles, the official Ranc - Disables the `embedded-cluster-api` feature in Rancher. - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. -The above webhooks can be removed through the Rancher UI as well and are accessed from the left-hand navigation from your local cluster under the **More Resources** > **Admission** section or found by using the **Resource Search** field and inputting the webhook names. Additionally, the following `kubectl` commands can also be used to remove the necessary webhooks: +These webhooks can be removed through the Rancher UI as well: + +1. In the upper left corner, click **☰** > **Cluster Management**. +1. Select your local cluster. +1. In the left-hand navigation menu, select **More Resources** > **Admission**. +1. From the dropdown you will see the Resource pages for `MutatingWebhookConfiguration` and `ValidatingWebhookConfiguration`, click through both pages to remove the necessary webhooks. +1. On the respective Resource pages, click the **⋮** that are attached to the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks to select the **Delete** option. Once deleted, you should see the webhooks removed immediately. + +They can also be accessed by entering the names of the webhooks into the **Resource Search** field. + +The following `kubectl` commands can manually remove the necessary webhooks: ```console kubectl delete mutatingwebhookconfiguration.admissionregistration.k8s.io mutating-webhook-configuration diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index 4ff36435ca5..970430093d8 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -14,9 +14,9 @@ Below is a visual representation of the key components of Rancher Turtles and th ## Security -[SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. +As defined by [Supply-chain Levels for Software Artifacts (SLSA)](https://slsa.dev/spec/v1.0/about), SLSA is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. -Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/security/slsa) document. +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements as an appropriate hardened build platform, with consistent build processes, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/security/slsa) document. ## Prerequisites @@ -27,7 +27,17 @@ To simplify setting up Rancher for installing Rancher Turtles, the official Ranc - Disables the `embedded-cluster-api` feature in Rancher. - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. -The above webhooks can be removed through the Rancher UI as well and are accessed from the left-hand navigation from your local cluster under the **More Resources** > **Admission** section or found by using the **Resource Search** field and inputting the webhook names. Additionally, the following `kubectl` commands can also be used to remove the necessary webhooks: +These webhooks can be removed through the Rancher UI as well: + +1. In the upper left corner, click **☰** > **Cluster Management**. +1. Select your local cluster. +1. In the left-hand navigation menu, select **More Resources** > **Admission**. +1. From the dropdown you will see the Resource pages for `MutatingWebhookConfiguration` and `ValidatingWebhookConfiguration`, click through both pages to remove the necessary webhooks. +1. On the respective Resource pages, click the **⋮** that are attached to the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks to select the **Delete** option. Once deleted, you should see the webhooks removed immediately. + +They can also be accessed by entering the names of the webhooks into the **Resource Search** field. + +The following `kubectl` commands can manually remove the necessary webhooks: ```console kubectl delete mutatingwebhookconfiguration.admissionregistration.k8s.io mutating-webhook-configuration diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index 4ff36435ca5..970430093d8 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -14,9 +14,9 @@ Below is a visual representation of the key components of Rancher Turtles and th ## Security -[SLSA](https://slsa.dev/spec/v1.0/about) is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. +As defined by [Supply-chain Levels for Software Artifacts (SLSA)](https://slsa.dev/spec/v1.0/about), SLSA is a set of incrementally adoptable guidelines for supply chain security, established by industry consensus. The specification set by SLSA is useful for both software producers and consumers: producers can follow SLSA’s guidelines to make their software supply chain more secure, and consumers can use SLSA to make decisions about whether to trust a software package. -Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements for appropriate build platform, consistent build process, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/security/slsa) document. +Rancher Turtles meets [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) requirements as an appropriate hardened build platform, with consistent build processes, and provenance distribution. For more information, visit the [Rancher Turtles Security](https://turtles.docs.rancher.com/security/slsa) document. ## Prerequisites @@ -27,7 +27,17 @@ To simplify setting up Rancher for installing Rancher Turtles, the official Ranc - Disables the `embedded-cluster-api` feature in Rancher. - Deletes the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks, as they are no longer needed. -The above webhooks can be removed through the Rancher UI as well and are accessed from the left-hand navigation from your local cluster under the **More Resources** > **Admission** section or found by using the **Resource Search** field and inputting the webhook names. Additionally, the following `kubectl` commands can also be used to remove the necessary webhooks: +These webhooks can be removed through the Rancher UI as well: + +1. In the upper left corner, click **☰** > **Cluster Management**. +1. Select your local cluster. +1. In the left-hand navigation menu, select **More Resources** > **Admission**. +1. From the dropdown you will see the Resource pages for `MutatingWebhookConfiguration` and `ValidatingWebhookConfiguration`, click through both pages to remove the necessary webhooks. +1. On the respective Resource pages, click the **⋮** that are attached to the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks to select the **Delete** option. Once deleted, you should see the webhooks removed immediately. + +They can also be accessed by entering the names of the webhooks into the **Resource Search** field. + +The following `kubectl` commands can manually remove the necessary webhooks: ```console kubectl delete mutatingwebhookconfiguration.admissionregistration.k8s.io mutating-webhook-configuration From c902e292225917d6c645b398d8d240b938f27cf5 Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Tue, 16 Apr 2024 08:17:26 -0700 Subject: [PATCH 53/67] Updating phrasing after PR review for added steps. Signed-off-by: Sunil Singh --- docs/integrations-in-rancher/cluster-api/overview.md | 6 +++--- .../integrations-in-rancher/cluster-api/overview.md | 6 +++--- .../integrations-in-rancher/cluster-api/overview.md | 6 +++--- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 970430093d8..0adb6deb5ac 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -32,10 +32,10 @@ These webhooks can be removed through the Rancher UI as well: 1. In the upper left corner, click **☰** > **Cluster Management**. 1. Select your local cluster. 1. In the left-hand navigation menu, select **More Resources** > **Admission**. -1. From the dropdown you will see the Resource pages for `MutatingWebhookConfiguration` and `ValidatingWebhookConfiguration`, click through both pages to remove the necessary webhooks. -1. On the respective Resource pages, click the **⋮** that are attached to the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks to select the **Delete** option. Once deleted, you should see the webhooks removed immediately. +1. From the dropdown, select the Resource pages for `MutatingWebhookConfiguration` and `ValidatingWebhookConfiguration`. +1. On the respective Resource pages, click the **⋮** that are attached to the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks and select the **Delete** option. -They can also be accessed by entering the names of the webhooks into the **Resource Search** field. +The webhooks can also be accessed by entering the names of the webhooks into the **Resource Search** field. The following `kubectl` commands can manually remove the necessary webhooks: diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index 970430093d8..0adb6deb5ac 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -32,10 +32,10 @@ These webhooks can be removed through the Rancher UI as well: 1. In the upper left corner, click **☰** > **Cluster Management**. 1. Select your local cluster. 1. In the left-hand navigation menu, select **More Resources** > **Admission**. -1. From the dropdown you will see the Resource pages for `MutatingWebhookConfiguration` and `ValidatingWebhookConfiguration`, click through both pages to remove the necessary webhooks. -1. On the respective Resource pages, click the **⋮** that are attached to the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks to select the **Delete** option. Once deleted, you should see the webhooks removed immediately. +1. From the dropdown, select the Resource pages for `MutatingWebhookConfiguration` and `ValidatingWebhookConfiguration`. +1. On the respective Resource pages, click the **⋮** that are attached to the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks and select the **Delete** option. -They can also be accessed by entering the names of the webhooks into the **Resource Search** field. +The webhooks can also be accessed by entering the names of the webhooks into the **Resource Search** field. The following `kubectl` commands can manually remove the necessary webhooks: diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index 970430093d8..0adb6deb5ac 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -32,10 +32,10 @@ These webhooks can be removed through the Rancher UI as well: 1. In the upper left corner, click **☰** > **Cluster Management**. 1. Select your local cluster. 1. In the left-hand navigation menu, select **More Resources** > **Admission**. -1. From the dropdown you will see the Resource pages for `MutatingWebhookConfiguration` and `ValidatingWebhookConfiguration`, click through both pages to remove the necessary webhooks. -1. On the respective Resource pages, click the **⋮** that are attached to the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks to select the **Delete** option. Once deleted, you should see the webhooks removed immediately. +1. From the dropdown, select the Resource pages for `MutatingWebhookConfiguration` and `ValidatingWebhookConfiguration`. +1. On the respective Resource pages, click the **⋮** that are attached to the `mutating-webhook-configuration` and `validating-webhook-configuration` webhooks and select the **Delete** option. -They can also be accessed by entering the names of the webhooks into the **Resource Search** field. +The webhooks can also be accessed by entering the names of the webhooks into the **Resource Search** field. The following `kubectl` commands can manually remove the necessary webhooks: From 679f210e157520e247715aedc73746e92309122d Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Tue, 16 Apr 2024 10:53:26 -0700 Subject: [PATCH 54/67] Replacing /docs links with latest. Signed-off-by: Sunil Singh --- docs/integrations-in-rancher/cluster-api/overview.md | 6 +++--- .../integrations-in-rancher/cluster-api/overview.md | 6 +++--- .../integrations-in-rancher/cluster-api/overview.md | 6 +++--- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/overview.md b/docs/integrations-in-rancher/cluster-api/overview.md index 0adb6deb5ac..a2878c3ab04 100644 --- a/docs/integrations-in-rancher/cluster-api/overview.md +++ b/docs/integrations-in-rancher/cluster-api/overview.md @@ -92,7 +92,7 @@ By adding the Turtles repository via the Rancher UI, Rancher can process the ins 1. Click **Rancher Turtles - the Cluster API Extension**. 1. Click **Install > Next > Install**. -This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). +This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values). The installation may take a few minutes and after completing you can see the following new deployments in the cluster: @@ -177,7 +177,7 @@ stringData: :::info -For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) +For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values) ::: @@ -185,7 +185,7 @@ For detailed information on the values supported by the chart and their usage, r :::note -Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. +Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. ::: diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md index 0adb6deb5ac..a2878c3ab04 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -92,7 +92,7 @@ By adding the Turtles repository via the Rancher UI, Rancher can process the ins 1. Click **Rancher Turtles - the Cluster API Extension**. 1. Click **Install > Next > Install**. -This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). +This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values). The installation may take a few minutes and after completing you can see the following new deployments in the cluster: @@ -177,7 +177,7 @@ stringData: :::info -For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) +For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values) ::: @@ -185,7 +185,7 @@ For detailed information on the values supported by the chart and their usage, r :::note -Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. +Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. ::: diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index 0adb6deb5ac..a2878c3ab04 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -92,7 +92,7 @@ By adding the Turtles repository via the Rancher UI, Rancher can process the ins 1. Click **Rancher Turtles - the Cluster API Extension**. 1. Click **Install > Next > Install**. -This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values). +This process uses the default values for the Helm chart, which are good for most installations. If your configuration requires overriding some of these defaults, you can either specify the values during installation from the Rancher UI or you can [manually install the chart via Helm](#installing-via-helm). For details about available values, see the Rancher Turtles [Helm chart reference guide](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values). The installation may take a few minutes and after completing you can see the following new deployments in the cluster: @@ -177,7 +177,7 @@ stringData: :::info -For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) +For detailed information on the values supported by the chart and their usage, refer to [Helm chart options](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values) ::: @@ -185,7 +185,7 @@ For detailed information on the values supported by the chart and their usage, r :::note -Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. +Remember that if you opt for this installation option, you must manage the CAPI Operator installation yourself. You can follow the [CAPI Operator guide](https://turtles.docs.rancher.com/tasks/capi-operator/intro) in the Rancher Turtles documentation for assistance. ::: From 59818ec88214a6f2dacdf02f029e2bdfa8833a9f Mon Sep 17 00:00:00 2001 From: Sunil Singh Date: Tue, 16 Apr 2024 10:59:13 -0700 Subject: [PATCH 55/67] Updating links on CAPI page. Signed-off-by: Sunil Singh --- docs/integrations-in-rancher/cluster-api/cluster-api.md | 2 +- .../integrations-in-rancher/cluster-api/cluster-api.md | 2 +- .../integrations-in-rancher/cluster-api/cluster-api.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/integrations-in-rancher/cluster-api/cluster-api.md b/docs/integrations-in-rancher/cluster-api/cluster-api.md index 7cb1dae14e6..32e124ba058 100644 --- a/docs/integrations-in-rancher/cluster-api/cluster-api.md +++ b/docs/integrations-in-rancher/cluster-api/cluster-api.md @@ -9,6 +9,6 @@ title: Cluster API (CAPI) with Rancher Turtles [Rancher Turtles](https://turtles.docs.rancher.com/) is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: - Import CAPI clusters into Rancher, by installing the Rancher Cluster Agent in CAPI provisioned clusters. -- Configure the [CAPI Operator](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values). +- Configure the [CAPI Operator](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values). The [Overview](./overview.md) section outlines installation options, Rancher Turtles architecture, and a brief demo. For more details, see the [Rancher Turtles documentation](https://turtles.docs.rancher.com/). diff --git a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/cluster-api.md b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/cluster-api.md index 7cb1dae14e6..32e124ba058 100644 --- a/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/cluster-api.md +++ b/versioned_docs/version-2.7/integrations-in-rancher/cluster-api/cluster-api.md @@ -9,6 +9,6 @@ title: Cluster API (CAPI) with Rancher Turtles [Rancher Turtles](https://turtles.docs.rancher.com/) is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: - Import CAPI clusters into Rancher, by installing the Rancher Cluster Agent in CAPI provisioned clusters. -- Configure the [CAPI Operator](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values). +- Configure the [CAPI Operator](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values). The [Overview](./overview.md) section outlines installation options, Rancher Turtles architecture, and a brief demo. For more details, see the [Rancher Turtles documentation](https://turtles.docs.rancher.com/). diff --git a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md index 7cb1dae14e6..32e124ba058 100644 --- a/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md +++ b/versioned_docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md @@ -9,6 +9,6 @@ title: Cluster API (CAPI) with Rancher Turtles [Rancher Turtles](https://turtles.docs.rancher.com/) is a [Rancher extension](../rancher-extensions.md) that manages the lifecycle of provisioned Kubernetes clusters, by providing integration between your Cluster API (CAPI) and Rancher. With Rancher Turtles, you can: - Import CAPI clusters into Rancher, by installing the Rancher Cluster Agent in CAPI provisioned clusters. -- Configure the [CAPI Operator](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values). +- Configure the [CAPI Operator](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values). The [Overview](./overview.md) section outlines installation options, Rancher Turtles architecture, and a brief demo. For more details, see the [Rancher Turtles documentation](https://turtles.docs.rancher.com/). From edd9f47033a5ae4137b27a9cfa279fa644beeb0c Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Mon, 22 Apr 2024 15:00:30 -0700 Subject: [PATCH 56/67] Add cattle cluster agent cpu and mem request recommendation --- .../rancher-managed-clusters/tips-to-set-up-containers.md | 7 +++++++ .../rancher-managed-clusters/tips-to-set-up-containers.md | 7 ++++++- .../rancher-managed-clusters/tips-to-set-up-containers.md | 7 ++++++- 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/docs/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md b/docs/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md index 0cec0a91864..343cb9fd151 100644 --- a/docs/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md +++ b/docs/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md @@ -21,6 +21,7 @@ Smaller distributions such as Alpine and BusyBox reduce container image size and Popular distributions such as Ubuntu, Fedora, and CentOS are more field-tested and offer more functionality. ### Start with a FROM scratch container + If your microservice is a standalone static binary, you should use a FROM scratch container. The FROM scratch container is an [official Docker image](https://hub.docker.com/_/scratch) that is empty so that you can use it to design minimal images. @@ -28,9 +29,11 @@ The FROM scratch container is an [official Docker image](https://hub.docker.com/ This will have the smallest attack surface and smallest image size. ### Run Container Processes as Unprivileged + When possible, use a non-privileged user when running processes within your container. While container runtimes provide isolation, vulnerabilities and attacks are still possible. Inadvertent or accidental host mounts can also be impacted if the container is running as root. For details on configuring a security context for a pod or container, refer to the [Kubernetes docs](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). ### Define Resource Limits + Apply CPU and memory limits to your pods. This can help manage the resources on your worker nodes and avoid a malfunctioning microservice from impacting other microservices. In standard Kubernetes, you can set resource limits on the namespace level. In Rancher, you can set resource limits on the project level and they will propagate to all the namespaces within the project. For details, refer to the Rancher docs. @@ -40,6 +43,7 @@ When setting resource quotas, if you set anything related to CPU or Memory (i.e. The Kubernetes docs have more information on how resource limits can be set at the [container level](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) and the namespace level. ### Define Resource Requirements + You should apply CPU and memory requirements to your pods. This is crucial for informing the scheduler which type of compute node your pod needs to be placed on, and ensuring it does not over-provision that node. In Kubernetes, you can set a resource requirement by defining `resources.requests` in the resource requests field in a pod's container spec. For details, refer to the [Kubernetes docs](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container). :::note @@ -50,7 +54,10 @@ If you set a resource limit for the namespace that the pod is deployed in, and t It is recommended to define resource requirements on the container level because otherwise, the scheduler makes assumptions that will likely not be helpful to your application when the cluster experiences load. +TODO we recommend to set your CPU request at 50m and memory request at 100Mi, however this is a base recommendation and customers should choose based on resources available in their environment and their use case on how they use Rancher. + ### Liveness and Readiness Probes + Set up liveness and readiness probes for your container. Unless your container completely crashes, Kubernetes will not know it's unhealthy unless you create an endpoint or mechanism that can report container status. Alternatively, make sure your container halts and crashes if unhealthy. The Kubernetes docs show how to [configure liveness and readiness probes for containers.](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) diff --git a/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md b/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md index 0cec0a91864..98dfae1ffaa 100644 --- a/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md +++ b/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md @@ -21,6 +21,7 @@ Smaller distributions such as Alpine and BusyBox reduce container image size and Popular distributions such as Ubuntu, Fedora, and CentOS are more field-tested and offer more functionality. ### Start with a FROM scratch container + If your microservice is a standalone static binary, you should use a FROM scratch container. The FROM scratch container is an [official Docker image](https://hub.docker.com/_/scratch) that is empty so that you can use it to design minimal images. @@ -28,9 +29,11 @@ The FROM scratch container is an [official Docker image](https://hub.docker.com/ This will have the smallest attack surface and smallest image size. ### Run Container Processes as Unprivileged + When possible, use a non-privileged user when running processes within your container. While container runtimes provide isolation, vulnerabilities and attacks are still possible. Inadvertent or accidental host mounts can also be impacted if the container is running as root. For details on configuring a security context for a pod or container, refer to the [Kubernetes docs](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). ### Define Resource Limits + Apply CPU and memory limits to your pods. This can help manage the resources on your worker nodes and avoid a malfunctioning microservice from impacting other microservices. In standard Kubernetes, you can set resource limits on the namespace level. In Rancher, you can set resource limits on the project level and they will propagate to all the namespaces within the project. For details, refer to the Rancher docs. @@ -40,6 +43,7 @@ When setting resource quotas, if you set anything related to CPU or Memory (i.e. The Kubernetes docs have more information on how resource limits can be set at the [container level](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) and the namespace level. ### Define Resource Requirements + You should apply CPU and memory requirements to your pods. This is crucial for informing the scheduler which type of compute node your pod needs to be placed on, and ensuring it does not over-provision that node. In Kubernetes, you can set a resource requirement by defining `resources.requests` in the resource requests field in a pod's container spec. For details, refer to the [Kubernetes docs](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container). :::note @@ -48,9 +52,10 @@ If you set a resource limit for the namespace that the pod is deployed in, and t ::: -It is recommended to define resource requirements on the container level because otherwise, the scheduler makes assumptions that will likely not be helpful to your application when the cluster experiences load. +As a starting point, we recommend setting your CPU request at 50m and memory request at 100Mi. You should choose your limits based on the resources available in your environment and your use case with Rancher. It is recommended to define resource requirements on the container level because otherwise, the scheduler makes assumptions that will likely not be helpful to your application when the cluster experiences load. ### Liveness and Readiness Probes + Set up liveness and readiness probes for your container. Unless your container completely crashes, Kubernetes will not know it's unhealthy unless you create an endpoint or mechanism that can report container status. Alternatively, make sure your container halts and crashes if unhealthy. The Kubernetes docs show how to [configure liveness and readiness probes for containers.](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) diff --git a/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md b/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md index 0cec0a91864..98dfae1ffaa 100644 --- a/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md +++ b/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md @@ -21,6 +21,7 @@ Smaller distributions such as Alpine and BusyBox reduce container image size and Popular distributions such as Ubuntu, Fedora, and CentOS are more field-tested and offer more functionality. ### Start with a FROM scratch container + If your microservice is a standalone static binary, you should use a FROM scratch container. The FROM scratch container is an [official Docker image](https://hub.docker.com/_/scratch) that is empty so that you can use it to design minimal images. @@ -28,9 +29,11 @@ The FROM scratch container is an [official Docker image](https://hub.docker.com/ This will have the smallest attack surface and smallest image size. ### Run Container Processes as Unprivileged + When possible, use a non-privileged user when running processes within your container. While container runtimes provide isolation, vulnerabilities and attacks are still possible. Inadvertent or accidental host mounts can also be impacted if the container is running as root. For details on configuring a security context for a pod or container, refer to the [Kubernetes docs](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). ### Define Resource Limits + Apply CPU and memory limits to your pods. This can help manage the resources on your worker nodes and avoid a malfunctioning microservice from impacting other microservices. In standard Kubernetes, you can set resource limits on the namespace level. In Rancher, you can set resource limits on the project level and they will propagate to all the namespaces within the project. For details, refer to the Rancher docs. @@ -40,6 +43,7 @@ When setting resource quotas, if you set anything related to CPU or Memory (i.e. The Kubernetes docs have more information on how resource limits can be set at the [container level](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) and the namespace level. ### Define Resource Requirements + You should apply CPU and memory requirements to your pods. This is crucial for informing the scheduler which type of compute node your pod needs to be placed on, and ensuring it does not over-provision that node. In Kubernetes, you can set a resource requirement by defining `resources.requests` in the resource requests field in a pod's container spec. For details, refer to the [Kubernetes docs](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container). :::note @@ -48,9 +52,10 @@ If you set a resource limit for the namespace that the pod is deployed in, and t ::: -It is recommended to define resource requirements on the container level because otherwise, the scheduler makes assumptions that will likely not be helpful to your application when the cluster experiences load. +As a starting point, we recommend setting your CPU request at 50m and memory request at 100Mi. You should choose your limits based on the resources available in your environment and your use case with Rancher. It is recommended to define resource requirements on the container level because otherwise, the scheduler makes assumptions that will likely not be helpful to your application when the cluster experiences load. ### Liveness and Readiness Probes + Set up liveness and readiness probes for your container. Unless your container completely crashes, Kubernetes will not know it's unhealthy unless you create an endpoint or mechanism that can report container status. Alternatively, make sure your container halts and crashes if unhealthy. The Kubernetes docs show how to [configure liveness and readiness probes for containers.](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) From 98bb32df49ccdf549bc43d56e51deb9b821b02a4 Mon Sep 17 00:00:00 2001 From: Jacie Date: Wed, 24 Apr 2024 10:25:32 +0800 Subject: [PATCH 57/67] update missing Chinese translation for v2.7 --- .../current/faq/general-faq.md | 50 +- .../helm-chart-options.md | 2 +- .../deploy-workloads/deploy-workloads.md | 4 + .../restore-rancher.md | 39 +- .../cluster-api/cluster-api.md | 2 +- .../cluster-api/overview.md | 57 +- .../reference-guides/rancher-webhook.md | 107 +- .../version-2.7.json | 140 +- .../version-2.7/faq/deprecated-features.md | 35 + .../version-2.7/faq/general-faq.md | 50 +- .../air-gapped-upgrades.md | 50 +- ...install-upgrade-on-a-kubernetes-cluster.md | 352 ++ .../rancher-on-aks.md | 12 +- .../rancher-on-amazon-eks.md | 22 +- .../rancher-on-gke.md | 20 +- .../rollbacks.md | 66 +- .../upgrades.md | 41 +- .../installation-and-upgrade.md | 98 + .../installation-references/feature-flags.md | 28 +- .../helm-chart-options.md | 118 +- .../installation-references.md | 9 + .../install-docker.md | 18 +- .../installation-requirements.md | 221 ++ .../port-requirements.md | 173 +- .../air-gapped-helm-cli-install.md | 36 + .../infrastructure-private-registry.md | 29 +- .../install-rancher-ha.md | 74 +- .../publish-images.md | 55 +- .../other-installation-methods.md | 23 + .../install-rancher.md | 10 +- .../rancher-behind-an-http-proxy.md | 17 + .../set-up-infrastructure.md | 10 +- .../certificate-troubleshooting.md | 18 +- .../rancher-on-a-single-node-with-docker.md | 214 ++ .../roll-back-docker-installed-rancher.md | 24 +- .../upgrade-docker-installed-rancher.md | 101 +- .../resources/add-tls-secrets.md | 6 +- .../resources/choose-a-rancher-version.md | 44 +- .../resources/helm-version-requirements.md | 8 +- .../resources/local-system-charts.md | 6 +- .../resources/resources.md | 29 + .../resources/update-rancher-certificate.md | 21 +- .../resources/upgrade-cert-manager.md | 17 +- .../upgrade-and-roll-back-kubernetes.md | 12 +- ...de-kubernetes-without-upgrading-rancher.md | 24 +- .../version-2.7/getting-started/overview.md | 24 +- .../deploy-rancher-manager/aws-marketplace.md | 10 +- .../deploy-rancher-manager/aws.md | 8 +- .../deploy-rancher-manager/azure.md | 8 +- .../deploy-rancher-manager.md | 24 + .../deploy-rancher-manager/digitalocean.md | 8 +- .../deploy-rancher-manager/equinix-metal.md | 12 +- .../deploy-rancher-manager/gcp.md | 8 +- .../deploy-rancher-manager/helm-cli.md | 18 +- .../deploy-rancher-manager/hetzner-cloud.md | 8 +- .../deploy-rancher-manager/linode.md | 83 + .../deploy-rancher-manager/outscale-qs.md | 8 +- .../deploy-rancher-manager/vagrant.md | 13 +- .../deploy-workloads/deploy-workloads.md | 12 + .../quick-start-guides/quick-start-guides.md | 21 + .../advanced-user-guides.md | 11 + .../cis-scan-guides/cis-scan-guides.md | 17 + ...le-api-audit-log-in-downstream-clusters.md | 140 + .../enable-experimental-features.md | 127 + .../istio-setup-guide/istio-setup-guide.md | 34 + .../manage-project-resource-quotas.md | 50 + .../manage-projects/manage-projects.md | 41 + .../monitoring-alerting-guides.md | 15 + .../enable-prometheus-federator.md | 4 +- .../prometheus-federator-guides.md | 12 + .../set-up-monitoring-for-workloads.md | 2 +- .../advanced-configuration.md | 19 + .../advanced-configuration/prometheusrules.md | 2 +- .../monitoring-v2-configuration-guides.md | 55 + .../about-provisioning-drivers.md | 51 + .../about-rke1-templates.md | 130 + .../authentication-config.md | 145 + .../configure-okta-saml.md | 2 +- ...on-permissions-and-global-configuration.md | 85 + ...re-microsoft-ad-federation-service-saml.md | 39 + .../configure-ms-adfs-for-rancher.md | 23 +- .../configure-rancher-for-ms-adfs.md | 25 +- .../configure-openldap/configure-openldap.md | 56 + .../openldap-config-reference.md | 67 +- .../about-group-permissions.md | 8 +- .../configure-shibboleth-saml.md | 104 + .../global-permissions.md | 134 +- .../manage-role-based-access-control-rbac.md | 29 + ...up-rancher-launched-kubernetes-clusters.md | 12 +- .../backup-restore-and-disaster-recovery.md | 100 + .../restore-rancher.md | 39 +- .../deploy-apps-across-clusters.md | 21 + .../helm-charts-in-rancher/create-apps.md | 13 +- .../helm-charts-in-rancher.md | 216 ++ .../infrastructure-setup.md | 13 + .../kubernetes-cluster-setup.md | 11 + ...checklist-for-production-ready-clusters.md | 52 + .../kubernetes-clusters-in-rancher-setup.md | 78 + .../migrate-to-out-of-tree-amazon.md | 198 + .../migrate-to-out-of-tree-vsphere.md | 115 + .../google-compute-engine.md | 2 +- .../set-up-cloud-providers.md | 47 + ...usters-from-hosted-kubernetes-providers.md | 33 + .../use-windows-clusters.md | 284 ++ .../horizontal-pod-autoscaler.md | 30 + .../kubernetes-resources-setup.md | 67 + .../ingress-configuration.md | 1 - .../load-balancer-and-ingress-controller.md | 65 + .../workloads-and-pods/deploy-workloads.md | 2 +- .../workloads-and-pods/workloads-and-pods.md | 82 + .../launch-kubernetes-with-rancher.md | 84 + .../nutanix/nutanix.md | 21 + .../use-new-nodes-in-an-infra-provider.md | 156 + ...rovision-kubernetes-clusters-in-vsphere.md | 4 +- .../vsphere/vsphere.md | 60 + .../access-clusters/access-clusters.md | 65 + .../authorized-cluster-endpoint.md | 13 +- .../use-kubectl-and-kubeconfig.md | 7 +- .../create-kubernetes-persistent-storage.md | 79 + .../install-cluster-autoscaler.md | 28 + .../manage-clusters/manage-clusters.md | 36 + .../manage-clusters/nodes-and-node-pools.md | 9 +- .../provisioning-storage-examples.md | 15 + .../new-user-guides/new-user-guides.md | 13 + .../cis-scans/cis-scans.md | 121 + .../aws-cloud-marketplace.md | 36 + .../aws-cloud-marketplace/install-adapter.md | 29 +- .../cloud-marketplace/cloud-marketplace.md | 11 + .../cluster-api/cluster-api.md | 14 + .../cluster-api/overview.md | 260 ++ .../elemental/elemental.md | 30 + .../fleet/architecture.md | 11 + .../integrations-in-rancher/fleet/fleet.md | 27 + .../integrations-in-rancher/fleet/overview.md | 66 + .../fleet/use-fleet-behind-a-proxy.md | 58 + .../fleet/windows-support.md | 25 + .../harvester/harvester.md | 15 + .../harvester/overview.md | 42 + .../integrations-in-rancher.mdx | 51 + .../configuration-options.md | 43 + .../integrations-in-rancher/istio/istio.md | 140 + .../kubernetes-distributions.md | 34 + .../kubewarden/kubewarden.md | 31 + .../custom-resource-configuration.md | 12 + .../logging/logging.md | 115 + .../longhorn/longhorn.md | 15 + .../longhorn/overview.md | 74 + .../built-in-dashboards.md | 6 +- .../monitoring-and-alerting.md | 106 + .../neuvector/neuvector.md | 27 + .../neuvector/overview.md | 196 + .../rancher-desktop.md | 33 + .../about-the-api/about-the-api.md | 93 + .../backup-configuration.md | 6 + .../backup-restore-configuration.md | 12 + .../best-practices/best-practices.md | 21 + .../monitoring-best-practices.md | 2 +- .../rancher-managed-clusters.md | 23 + .../rancher-server/rancher-server.md | 21 + ...and-best-practices-for-rancher-at-scale.md | 117 + .../cli-with-rancher/cli-with-rancher.md | 9 + .../cluster-configuration.md | 32 + .../downstream-cluster-configuration.md | 9 + .../machine-configuration.md | 9 + .../node-template-configuration.md | 9 + .../node-template-configuration/vsphere.md | 4 +- .../gke-cluster-configuration.md | 324 ++ .../gke-private-clusters.md | 2 +- .../rancher-server-configuration.md | 16 + .../rke2-cluster-configuration.md | 2 +- .../use-existing-nodes/use-existing-nodes.md | 141 + .../monitoring-v2-configuration.md | 15 + .../monitoring-v2-configuration/routes.md | 2 +- .../prometheus-federator.md | 109 + .../rancher-manager-architecture.md | 21 + .../hardening-guides/hardening-guides.md | 55 + .../k3s-hardening-guide.md | 744 ++++ ...sessment-guide-with-cis-v1.23-k8s-v1.23.md | 34 +- ...sessment-guide-with-cis-v1.24-k8s-v1.24.md | 3208 ++++++++++++++++ ...ide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md | 3215 +++++++++++++++++ .../rke1-hardening-guide.md | 515 +++ ...sessment-guide-with-cis-v1.23-k8s-v1.23.md | 26 +- ...sessment-guide-with-cis-v1.24-k8s-v1.24.md | 3046 ++++++++++++++++ ...ide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md | 2862 +++++++++++++++ .../rke2-hardening-guide.md | 273 ++ ...sessment-guide-with-cis-v1.23-k8s-v1.23.md | 26 +- ...sessment-guide-with-cis-v1.24-k8s-v1.24.md | 3202 ++++++++++++++++ ...ide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md | 2967 +++++++++++++++ .../rancher-security-best-practices.md | 21 + .../rancher-security/rancher-security.md | 95 + .../security-advisories-and-cves.md | 14 +- .../selinux-rpm/selinux-rpm.md | 20 + .../reference-guides/rancher-webhook.md | 116 +- .../single-node-rancher-in-docker.md | 9 + .../user-settings/user-settings.md | 19 + .../_cluster-capabilities-table.md | 10 +- .../kubernetes-components.md | 21 + .../troubleshooting-etcd-nodes.md | 128 +- .../version-2.8/faq/general-faq.md | 50 +- .../helm-chart-options.md | 2 +- .../restore-rancher.md | 39 +- .../cluster-api/cluster-api.md | 2 +- .../cluster-api/overview.md | 57 +- .../reference-guides/rancher-webhook.md | 107 +- 204 files changed, 28748 insertions(+), 1017 deletions(-) create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/faq/deprecated-features.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-and-upgrade.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-references/installation-references.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/other-installation-methods.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/resources.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/linode.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/quick-start-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/advanced-user-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/monitoring-alerting-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/prometheus-federator-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/advanced-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/about-provisioning-drivers.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-permissions-and-global-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-microsoft-ad-federation-service-saml/configure-microsoft-ad-federation-service-saml.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-openldap/configure-openldap.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-shibboleth-saml/configure-shibboleth-saml.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-amazon.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/install-cluster-autoscaler/install-cluster-autoscaler.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/manage-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/provisioning-storage-examples.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/new-user-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cis-scans/cis-scans.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/aws-cloud-marketplace.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cloud-marketplace/cloud-marketplace.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cluster-api/cluster-api.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cluster-api/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/elemental/elemental.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/architecture.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/fleet.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/windows-support.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/harvester/harvester.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/harvester/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/integrations-in-rancher.mdx create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/istio/configuration-options/configuration-options.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/istio/istio.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/kubernetes-distributions/kubernetes-distributions.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/kubewarden/kubewarden.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/logging/custom-resource-configuration/custom-resource-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/logging/logging.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/longhorn/longhorn.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/longhorn/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/neuvector/neuvector.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/neuvector/overview.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/rancher-desktop.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/about-the-api/about-the-api.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/backup-restore-configuration/backup-restore-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/best-practices.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-server/rancher-server.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cli-with-rancher/cli-with-rancher.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/cluster-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/downstream-cluster-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/machine-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/node-template-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/rancher-server-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/prometheus-federator/prometheus-federator.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/hardening-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/rancher-security-best-practices.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/rancher-security.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/selinux-rpm/selinux-rpm.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/single-node-rancher-in-docker/single-node-rancher-in-docker.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/user-settings/user-settings.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.7/troubleshooting/kubernetes-components/kubernetes-components.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/faq/general-faq.md b/i18n/zh/docusaurus-plugin-content-docs/current/faq/general-faq.md index 45d4ae835ae..5cf116534af 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/faq/general-faq.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/faq/general-faq.md @@ -2,68 +2,50 @@ title: 一般常见问题解答 --- + + + + 本文包含了用户常见的 Rancher 2.x 问题。 有关常见技术问题，请参阅[常见技术问题解答](technical-items.md)。 -
- -**Rancher 2.x 支持 Docker Swarm 和 Mesos 作为环境类型吗？** +## Rancher 2.x 支持 Docker Swarm 和 Mesos 作为环境类型吗？ 如果你在 Rancher 2.x 中创建环境，Swarm 和 Mesos 将不再是可选的标准选项。但是，Swarm 和 Mesos 还能继续作为可以部署的商店应用程序。这是一个艰难的决定，但这是大势所趋。比如说，15,000 多个集群可能只有大约 200 个在运行 Swarm。 -
+## 是否可以使用 Rancher 2.x 管理 Azure Kubernetes 服务？ -**是否可以使用 Rancher 2.x 管理 Azure Kubernetes 服务？** +是的。请参阅我们的[集群管理]((../how-to-guides/new-user-guides/manage-clusters/manage-clusters.md))指南，了解 AKS 上可用的 Rancher 功能，以及相关的 [AKS 的文档](../getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md) -是的。 +## Rancher 是否支持 Windows？ -
+Rancher 支持 Windows Server 1809 容器。有关如何使用 Windows Worker 节点设置集群的详细信息，请参阅[为 Windows 配置自定义集群](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)。 -**Rancher 是否支持 Windows？** - -Rancher 支持 Windows Server 1809 容器。有关如何使用 Windows Worker 节点设置集群的详细信息，请参阅[为 Windows 配置自定义集群](../pages-for-subheaders/use-windows-clusters.md)。 - -
- -**Rancher 是否支持 Istio？** +## Rancher 是否支持 Istio？ Rancher 支持 [Istio](../pages-for-subheaders/istio.md)。 -此外，Istio 是在我们的微型 PaaS “Rio” 中实现的，它可以运行在 Rancher 2.x 以及任何符合 CNCF 的 Kubernetes 集群上。详情请参阅[这里](https://rio.io/)。 - -
- -**Rancher 2.x 是否支持使用 Hashicorp 的 Vault 来存储密文？** +## Rancher 2.x 是否支持使用 Hashicorp 的 Vault 来存储密文？ 密文管理已在我们的 roadmap 上，但我们尚未将该功能分配给特定版本。 -
- -**Rancher 2.x 是否也支持 RKT 容器？** +## Rancher 2.x 是否也支持 RKT 容器？ 目前，我们只支持 Docker。 -
- -**Rancher 2.x 是否支持将 Calico、Contiv、Contrail、Flannel、Weave net 等网络插件用于嵌入和已注册的 Kubernetes？** +## Rancher 2.x 是否支持将 Calico、Contiv、Contrail、Flannel、Weave net 等网络插件用于嵌入和已注册的 Kubernetes？ Rancher 开箱即用地为 Kubernetes 集群提供了几个 CNI 网络插件，分别是 Canal、Flannel、Calico 和 Weave。有关官方支持的详细信息，请参阅 [Rancher 支持矩阵](https://rancher.com/support-maintenance-terms/)。 -
- -**Rancher 是否计划支持 Traefik？** +## Rancher 是否计划支持 Traefik？ 目前，我们不打算提供嵌入式 Traefik 支持，但我们仍在探索负载均衡方案。 -
- -**我可以将 OpenShift Kubernetes 集群导入 2.x 吗？** +## 我可以将 OpenShift Kubernetes 集群导入 2.x 吗？ 我们的目标是运行任何上游 Kubernetes 集群。因此，Rancher 2.x 应该可以与 OpenShift 一起使用，但我们尚未对此进行测试。 -
- -**Rancher 会集成 Longhorn 吗？** +## Rancher 会集成 Longhorn 吗？ 是的。Longhorn 已集成到 Rancher 2.5+ 中。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index c924cc3c797..ca0cdcc3a2f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -159,7 +159,7 @@ Rancher 的一些功能（Helm Chart）需要使用互联网才能使用。你 不包括敏感数据时，可以使用 `proxy` 或 `extraEnv` Chart 选项。使用 `extraEnv` 时将忽略 `noProxy` Helm 选项。因此，`NO_PROXY` 环境变量也必须设置为 `extraEnv`。 -以下是使用 `extraEnv` Chart 选项设置代理的示例： +以下是使用 `proxy` Chart 选项设置代理的示例： ```plain --set proxy="http:///" diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md index 80c8505d35e..b0aa55e4fb4 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md @@ -2,6 +2,10 @@ title: 部署工作负载 --- + + + + 这些指南指导你完成一个应用的部署，包括如何将应用暴露在集群之外使用。 - [部署带有 Ingress 的工作负载](workload-ingress.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md index 11760970b23..eeb6cb521f0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md @@ -4,38 +4,23 @@ title: 还原 Rancher 本页概述了如何使用 Rancher 执行恢复。 -:::note 重要提示： +在以下情况下，请按照本页中的说明进行操作： +- 正在运行的 Rancher 实例与备份时的版本相同。 +- 上游（本地）集群与备份的位置相同。 -* 请按照此页面上的说明在已备份的同一集群上还原 Rancher。要把 Rancher 迁移到新集群，请参照步骤[迁移 Rancher](migrate-rancher-to-new-cluster.md)。 -* 在使用相同设置还原 Rancher 时，operator 将在还原开始时缩减 Rancher deployment，还原完成后又会扩展 deployment。因此，Rancher 在还原期间将不可用。 +:::note 重要提示 + +在使用相同设置还原 Rancher 时，operator 将在还原开始时缩减 Rancher deployment，还原完成后又会扩展 deployment。因此，Rancher 在还原期间将不可用。 + +::: + +:::tip + +* 按照以下步骤[迁移 Rancher](migrate-rancher-to-new-cluster.md)。 * 如果你需要在升级后将 Rancher 还原到先前版本，请参见[回滚](../../../getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md)。 ::: -## 使用 Rancher 2.6.4+ 进行回滚的其他步骤 - -Rancher v2.6.4 将 cluster-api 模块从 v0.4.4 升级到 v1.0.2。反过来，cluster-api 的 v1.0.2 版本将集群 API 的自定义资源定义 (CRD) 从 `cluster.x-k8s.io/v1alpha4` 升级到 `cluster.x-k8s.io/v1beta1`。当你尝试将 Rancher v2.6.4 回滚到以前版本的 Rancher v2.6.x 时，CRD 升级到 v1beta1 会导致回滚失败。这是因为使用旧 apiVersion (v1alpha4) 的 CRD 与 v1beta1 不兼容。 - -要避免回滚失败，你需要在尝试恢复操作或回滚**之前**运行以下 Rancher 脚本： - -* `verify.sh`：检查集群中是否有任何与 Rancher 相关的资源。 -* `cleanup.sh`：清理集群。 - -有关详细信息和源代码，请参阅 [rancher/rancher-cleanup repo](https://github.com/rancher/rancher-cleanup)。 - -:::caution - -`cleanup.sh` 运行的时候会有停机时间，这是因为脚本会删除 Rancher 创建的资源。 - -::: - -### 从 v2.6.4+ 回滚到较低版本的 v2.6.x - -1. 按照[说明](https://github.com/rancher/rancher-cleanup/blob/main/README.md)运行脚本。 -1. 按照[说明](../backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)在现有集群上安装 rancher-backup Helm Chart 并恢复之前的状态。 - 1. 省略步骤 3。 - 1. 执行到步骤 4 时，在要回滚到的 local 集群上安装 Rancher 2.6.x 版本。 - ### 创建 Restore 自定义资源 还原是通过创建 Restore 自定义资源实现的。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/cluster-api.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/cluster-api.md index aa10ce27651..1bd4160152e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/cluster-api.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/cluster-api.md @@ -9,6 +9,6 @@ title: Cluster API (CAPI) 与 Rancher Turtles [Rancher Turtles](https://turtles.docs.rancher.com/) 是一个 [Rancher 扩展](../rancher-extensions.md)，通过提供 Cluster API (CAPI) 和 Rancher 之间的集成来管理配置的 Kubernetes 集群的生命周期。使用 Rancher Turtles，你可以： - 通过在 CAPI 配置的集群中安装 Rancher Cluster Agent，将 CAPI 集群导入 Rancher。 -- 配置 [CAPI Operator](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values)。 +- 配置 [CAPI Operator](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values)。 [概述](./overview.md)部分介绍了安装选项、Rancher Turtles 架构和简要 Demo。有关详细信息，请参阅 [Rancher Turtles 文档](https://turtles.docs.rancher.com/)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/overview.md b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/overview.md index 2981c2199ef..5da4470bfd6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/integrations-in-rancher/cluster-api/overview.md @@ -12,6 +12,12 @@ title: 概述 ![overview](/img/30000ft_view.png) +## 安全 + +[SLSA](https://slsa.dev/spec/v1.0/about) 是一套由行业共识制定的可逐步采用的供应链安全指南。SLSA 制定的规范对软件生产者和消费者都很有用：生产者可以遵循 SLSA 的指导方针，使他们的软件供应链更加安全，消费者可以使用 SLSA 来决定是否信任软件包。 + +Rancher Turtles 满足 [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) 对适当的构建平台、一致的构建过程和来源分布的要求。更多信息请参阅 [Rancher Turtles 安全](https://turtles.docs.rancher.com/security/slsa)文档。 + ## 先决条件 在 Rancher 环境中安装 Rancher Turtles 之前，你必须禁用 Rancher 的 `embedded-cluster-api` 功能。这还包括清理 Rancher 专用的 webhook，否则这些 webhook 将与 CAPI 的 webhook 冲突。 @@ -21,6 +27,45 @@ title: 概述 - 禁用 Rancher 中的 `embedded-cluster-api` 功能。 - 删除不再需要的 `mutating-webhook-configuration` 和 `validating-webhook-configuration` webhook。 +这些 webhook 也可以通过 Rancher UI 删除： + +1. 点击左上角 **☰ > 集群管理**。 +1. 选择你的 local 集群。 +1. 在左侧导航菜单，选择 **More Resources** > **Admission**。 +1. 在下拉菜单中，选择资源页面的 `MutatingWebhookConfiguration` 和 `ValidatingWebhookConfiguration`。 +1. 在相应的资源页面上，点击 `mutating-webhook-configuration` and `validating-webhook-configuration` 后面的 **⋮** 然后选择 **删除**。 + +还可以通过在 **Resource Search** 字段中输入 webhook 的名称来访问到具体的 webhook。 + +以下的 `kubectl` 命令可以手动删除必要的 webhook： + +```console +kubectl delete mutatingwebhookconfiguration.admissionregistration.k8s.io mutating-webhook-configuration +``` + +```console +kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io validating-webhook-configuration +``` + +使用以下示例从控制台禁用 `embedded-cluster-api` 功能： + +1. 创建一个 `feature.yaml` 文件，将 `embedded-cluster-api` 设置为 false： + +```yaml title="feature.yaml" +apiVersion: management.cattle.io/v3 +kind: Feature +metadata: + name: embedded-cluster-api +spec: + value: false +``` + +2. 使用 `kubectl` 将 `feature.yaml` 文件应用到集群： + +```bash +kubectl apply -f feature.yaml +``` + ## 安装 Rancher Turtles Operator 你可以通过 Rancher UI 或使用 Helm 安装 Rancher Turtles operator。对于大多数环境推荐使用第一种方法。 @@ -47,7 +92,7 @@ title: 概述 1. 点击 **Rancher Turtles - the Cluster API Extension**。 1. 点击 **Install > Next > Install**. -此过程使用 Helm chart 的默认值，这些值适用于大部分的安装场景。如果你的配置需要覆盖其中一些默认值，你可以在安装期间通过 Rancher UI 指定这些值，也可以通过 [Helm 手动安装 Chart](#通过-helm-安装)。有关可用的 values 设置的详细信息，请参阅 Rancher Turtles 的 [Helm chart 参考指南](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values)。 +此过程使用 Helm chart 的默认值，这些值适用于大部分的安装场景。如果你的配置需要覆盖其中一些默认值，你可以在安装期间通过 Rancher UI 指定这些值，也可以通过 [Helm 手动安装 Chart](#通过-helm-安装)。有关可用的 values 设置的详细信息，请参阅 Rancher Turtles 的 [Helm chart 参考指南](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values)。 安装可能需要几分钟时间，安装完成后，你可以在集群中看到以下新部署： @@ -132,7 +177,7 @@ stringData: :::info -有关 chart 支持的 values 及其用法的详细信息，请参阅 [Helm chart 选项](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) +有关 chart 支持的 values 及其用法的详细信息，请参阅 [Helm chart 选项](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values) ::: @@ -140,7 +185,7 @@ stringData: :::note -请记住，如果使用此安装选项，你必须自行管理 CAPI Operator 的安装。你可以参照 Rancher Turtles 文档中的 [CAPI Operator 指南](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) +请记住，如果使用此安装选项，你必须自行管理 CAPI Operator 的安装。你可以参照 Rancher Turtles 文档中的 [CAPI Operator 指南](https://turtles.docs.rancher.com/tasks/capi-operator/intro) ::: @@ -213,9 +258,3 @@ spec: ```bash kubectl apply -f feature.yaml ``` - -## 安全 - -[SLSA](https://slsa.dev/spec/v1.0/about) 是一套由行业共识制定的可逐步采用的供应链安全指南。SLSA 制定的规范对软件生产者和消费者都很有用：生产者可以遵循 SLSA 的指导方针，使他们的软件供应链更加安全，消费者可以使用 SLSA 来决定是否信任软件包。 - -Rancher Turtles 满足 [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) 对适当的构建平台、一致的构建过程和来源分布的要求。更多信息请参阅 [Rancher Turtles 安全](https://turtles.docs.rancher.com/docs/security/slsa)文档。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-webhook.md b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-webhook.md index caa0335ca2a..e8a77888233 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-webhook.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/reference-guides/rancher-webhook.md @@ -2,10 +2,28 @@ title: Rancher Webhook --- + + + + Rancher-Webhook 是 Rancher 的重要组件，它与 Kubernetes 结合使用，用于增强安全性并为 Rancher 管理的集群启用关键功能。 如 [Kubernetes 文档](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/)中所述，它与 Kubernetes 的可扩展准入控制器集成，允许 Rancher-Webhook 检查发送到 Kubernetes API Server 的特定请求，添加自定义和 Rancher 相关的验证，以及 Rancher 相关请求的变化。Rancher-Webhook 使用 `rancher.cattle.io` `ValidatingWebhookConfiguration` 和 `rancher.cattle.io` `MutatingWebhookConfiguration` 管理要验证的资源，并覆盖任何手动编辑。 -Rancher 将 Rancher-Webhook 作为单独的 deployment 和服务部署在 local 和下游集群中。Rancher 使用 Helm 管理 Rancher-Webhook。需要注意的是，Rancher 可能会覆盖用户对 Helm 版本所做的修改。 + +Rancher 将 Rancher-Webhook 作为单独的 deployment 和服务部署在 local 和下游集群中。Rancher 使用 Helm 管理 Rancher-Webhook。需要注意的是，Rancher 可能会覆盖用户对 Helm 版本所做的修改。要安全地修改这些值，请参阅[自定义 Rancher-Webhook 配置](#自定义-rancher-webhook-配置) + +每个 Rancher 版本都设计为与某个具体版本的 Webhook 兼容，为方便起见，下面提供了各版本的兼容列表。 + +**注意：** Rancher 负责管理 webhook 的部署和升级。在多数情况下，不需要用户干预来确保 webhook 版本与你正在运行的 Rancher 版本兼容。 + + + +| Rancher Version | Webhook Version | Availability in Prime | Availability in Community | +| --------------- | --------------- | --------------------- | ------------------------- | +| v2.8.3 | v0.4.3 | ✓ | ✓ | +| v2.8.2 | v0.4.2 | ✓ | ✓ | +| v2.8.1 | v0.4.2 | ✓ | ✓ | +| v2.8.0 | v0.4.2 | ✗ | ✓ | ## 为什么我们需要它？ @@ -13,6 +31,58 @@ Rancher-Webhook 对于让 Rancher 保护集群免受恶意攻击并启用各种 Rancher 依赖 Rancher-Webhook 作为其功能的组成部分。如果没有 Webhook，Rancher 将不是一个完整的产品。 它为 Rancher 管理的集群提供了必要的保护，防止安全漏洞并确保集群的一致性和稳定性。 +## Webhook 验证哪些资源? + +你可以在 [webhook 仓库](https://github.com/rancher/webhook/blob/release/v0.4/docs.md)中找到 webhook 当前验证的资源列表。这些文档按组/版本（顶级标题）和资源（下一级标题）进行组织。可以通过查看与特定版本标签关联的 `docs.md` 文件来找到特定于一个版本的检查。请注意，`v0.3.6` 之前的 webhook 版本没有此文件。 + +## 绕过 Webhook + +有时，你必须绕过 Rancher 的 webhook 验证才能执行紧急还原操作或修复其他关键问题。避开操作是彻底的，这意味着在使用它时不会应用任何 webhook 验证或更改。不可能指定绕过某些验证，而让其他验证仍然可用。它们要么全部被绕过，要么全部处于活动状态。 + +:::danger + +Rancher 的 webhook 提供关键的安全保护。只有在所有其他选项都用尽之后，管理员才需要在特定情况下绕过 webhook。此外，应仔细控制绕过 webhook 的权限，切勿将该权限授予非管理员用户。 + +::: + +要绕过 webhook，请模拟 `rancher-webhook-sudo` 服务账号和 `system:masters` 组（两者都是必需的）： + +```bash +kubectl create -f example.yaml --as=system:serviceaccount:cattle-system:rancher-webhook-sudo --as-group=system:masters +``` + +## 自定义 Rancher-Webhook 配置 + +你可以通过 Helm 安装 Rancher-Webhook 时添加自定义 Helm values。在 Rancher-Webhook chart 的安装过程中，Rancher 会检查自定义的 Helm values。这些自定义值必须在 `cattle-system` 命名空间中，名称为 `rancher-config` 的 ConfigMap 的 data 属性下，增加 `rancher-webhook` 的配置定义。此键的值必须是有效的 YAML。 + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: rancher-config + namespace: cattle-system + labels: + app.kubernetes.io/part-of: "rancher" +data: + rancher-webhook: '{"port": 9553, "priorityClassName": "system-node-critical"}' +``` + +Rancher 会在检测到对 ConfigMap 值的更改时重新部署 Rancher-Webhook Chart。 + +### 在 Rancher 安装过程中自定义 Rancher-Webhook + +使用 Helm 安装 Rancher chart 时，可以在 local 集群中将自定义的 Helm values 添加到 Rancher-Webhook。Rancher-Webhook Chart 中的所有值都可以通过 `webhook` 名称下的嵌套变量访问。 + +这些值在安装过程中会同步到 `rancher-config` ConfigMap 中。 + +```bash +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + ... + --set webhook.port=9553 \ + --set webhook.priorityClassName="system-node-critical" +``` + ## 常见问题 ### 带有 Calico CNI 的 EKS 集群 @@ -24,8 +94,11 @@ Rancher 依赖 Rancher-Webhook 作为其功能的组成部分。如果没有 Web helm repo add rancher-charts https://charts.rancher.io helm upgrade --reuse-values rancher-webhook rancher-chart/rancher-webhook -n cattle-system --set global.hostNetwork=true ``` + **注意**：这个临时解决方法可能会违反环境的安全策略。此解决方法还要求主机网络上未使用端口 9443。 +**注意：** 默认情况下，Helm 使用 secrets。这是某些 webhook 版本验证用于存储信息的数据类型。在这种情况下，请使用 kubectl 更新 deployment 设置 `hostNetwork=true`，然后按照上述配置更新 webhook。 + ### 私有 GKE 集群 使用私有 GKE 集群时可能会发生错误，导致 Kubernetes API Server 无法与 Webhook 通信。以下错误消息可能会出现： @@ -33,4 +106,36 @@ helm upgrade --reuse-values rancher-webhook rancher-chart/rancher-webhook -n ca ``` Internal error occurred: failed calling webhook "rancher.cattle.io.namespaces.create-non-kubesystem": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s": context deadline exceeded ``` + 出现此问题的原因是防火墙规则限制了 API Server 与私有集群之间的通信。要解决此通信问题，你必须通过添加防火墙规则来允许 GKE Control Plane 通过端口 9443 与 Rancher-Webhook 进行通信。请参阅 [GKE 文档](https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules)，了解更新防火墙规则的详细信息和步骤。 + +### 由于 rancher-webhook 阻止访问导致应用部署失败 + +webhook 在 [namespaces](https://github.com/rancher/webhook/blob/release/v0.4/docs.md#psa-label-validation) 上提供额外的验证。其中一项验证可确保用户只有在具有适当权限的情况下才能更新 PSA 相关标签（`updatepsa` for `projects` in `management.cattle.io`）。这可能导致特定 operator（如 Tigera 或 Trident）在尝试部署带有 PSA 标签的命名空间时失败。有几种方法可以解决此问题： + +- 将应用程序配置为创建没有 PSA 标签的命名空间。如果用户希望将 PSA 应用于这些命名空间，则可以在配置后将它们添加到具有所需 PSA 的项目中。请参阅[设置 PSS 和 PSA 资源的文档](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/pod-security-standards.md)获取更具体的操作方法。 + - 这是首选选项，但并非所有应用程序都可以以这种方式进行配置。 +- 手动授予操作员管理命名空间下的 PSA 的权限。 + - 此选项将引入安全风险，因为运营商现在将能够为其有权访问的命名空间设置 PSA。这可能允许操作员部署特权 Pod，或通过其他方式实现集群接管。 +- 具有适当权限的用户帐户可以使用适当的配置预先创建命名空间。 + - 此选项取决于应用程序处理现有资源的能力。 + +## 特定版本的问题 + +**注意：** 以下是影响特定 Rancher/webhook 版本的高严重性问题的不完整列表。在大多数情况下，这些问题可以通过升级到更新的 Rancher 版本来解决。 + +### 回滚到不兼容的 Webhook 版本 + +**注意：** 这会影响回滚到 Rancher v2.7.5 或更早版本。 + +如果回滚到 Rancher v2.7.5 或更早版本，您可能会看到 webhook 版本太新，无法与运行 v2.7.5 之前版本的 Rancher 的下游集群兼容。这可能会导致各种不兼容问题。例如，项目成员可能无法创建命名空间。此外，当您回滚到下游集群中安装 webhook 之前的版本时，webhook 可能仍保持安装状态，这会导致类似的不兼容问题。 + +为了帮助缓解这些问题，您可以在回滚后运行 [adjust-downstream-webhook](https://github.com/rancherlabs/support-tools/tree/master/adjust-downstream-webhook) shell 脚本。该脚本为相应的 Rancher 版本选择并安装正确的 webhook 版本（或完全删除 webhook）。 + +### 项目用户无法创建命名空间 + +**注意：** 以下内容影响 Rancher v2.7.2 - v2.7.4。 + +项目用户可能无法在项目中创建命名空间，这包括项目所有者。此问题是由于 Rancher 自动将 webhook 升级到与当前安装的 Rancher 版本更新的版本不兼容而导致的。 + +为了帮助缓解这些问题，您可以在回滚后运行 [adjust-downstream-webhook](https://github.com/rancherlabs/support-tools/tree/master/adjust-downstream-webhook) shell 脚本。该脚本为相应的 Rancher 版本选择并安装正确的 webhook 版本（或完全删除 webhook）。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7.json b/i18n/zh/docusaurus-plugin-content-docs/version-2.7.json index 0a8cf301cf7..942da86f2c3 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7.json +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7.json @@ -11,13 +11,13 @@ "message": "快速入门指南", "description": "The label for category Quick Start Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Deploy Rancher": { + "sidebar.tutorialSidebar.category.Deploying Rancher Server": { "message": "部署 Rancher", - "description": "The label for category Deploy Rancher in sidebar tutorialSidebar" + "description": "The label for category Deploying Rancher Server in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Deploy Workloads": { + "sidebar.tutorialSidebar.category.Deploying Workloads": { "message": "部署工作负载", - "description": "The label for category Deploy Workloads in sidebar tutorialSidebar" + "description": "The label for category Deploying Workloads in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Installation and Upgrade": { "message": "安装和升级", @@ -64,26 +64,26 @@ "description": "The label for category New User Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Authentication, Permissions, and Global Configuration": { - "message": "身份验证、权限和全局配置", + "message": "认证、权限和全局配置", "description": "The label for category Authentication, Permissions, and Global Configuration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Authentication Config": { - "message": "身份验证配置", + "sidebar.tutorialSidebar.category.Configuring Authentication": { + "message": "配置认证", "description": "The label for category Authentication Config in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Configure OpenLDAP": { + "sidebar.tutorialSidebar.category.Configuring OpenLDAP": { "message": "配置 OpenLDAP", "description": "The label for category Configure OpenLDAP in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Configure Microsoft AD Federation Service (SAML)": { + "sidebar.tutorialSidebar.category.Configuring Microsoft AD Federation Service (SAML)": { "message": "配置 Microsoft AD FS (SAML)", "description": "The label for category Configure Microsoft AD Federation Service (SAML) in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Configure Shibboleth (SAML)": { + "sidebar.tutorialSidebar.category.Configuring Shibboleth (SAML)": { "message": "配置 Shibboleth (SAML)", "description": "The label for category Configure Shibboleth (SAML) in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Manage Role-Based Access Control (RBAC)": { + "sidebar.tutorialSidebar.category.Managing Role-Based Access Control (RBAC)": { "message": "管理 RBAC", "description": "The label for category Manage Role-Based Access Control (RBAC) in sidebar tutorialSidebar" }, @@ -95,9 +95,9 @@ "message": "关于 RKE1 模板", "description": "The label for category About RKE1 Templates in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Manage Clusters": { - "message": "管理集群", - "description": "The label for category Manage Clusters in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Cluster Administration": { + "message": "集群管理", + "description": "The label for category Cluster Administration in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Access Clusters": { "message": "访问集群", @@ -115,49 +115,57 @@ "message": "配置存储示例", "description": "The label for category Provisioning Storage Examples in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Kubernetes Cluster Setup": { + "sidebar.tutorialSidebar.category.Setting up a Kubernetes Cluster for Rancher Server": { "message": "Kubernetes 集群设置", - "description": "The label for category Kubernetes Cluster Setup in sidebar tutorialSidebar" + "description": "The label for category Setting up a Kubernetes Cluster for Rancher Server in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Infrastructure Setup": { "message": "基础设施设置", "description": "The label for category Infrastructure Setup in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Kubernetes Clusters in Rancher Setup": { - "message": "Rancher 设置中的 Kubernetes 集群", + "message": "Rancher 中的 Kubernetes 集群设置", "description": "The label for category Kubernetes Clusters in Rancher Setup in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Checklist for Production-Ready Clusters": { "message": "生产就绪集群检查清单", "description": "The label for category Checklist for Production-Ready Clusters in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Set Up Clusters from Hosted Kubernetes Providers": { + "sidebar.tutorialSidebar.category.Setting up Clusters from Hosted Kubernetes Providers": { "message": "通过托管 Kubernetes 提供商设置集群", - "description": "The label for category Set Up Clusters from Hosted Kubernetes Providers in sidebar tutorialSidebar" + "description": "The label for category Setting Up Clusters from Hosted Kubernetes Providers in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Launching Kubernetes on Windows Clusters": { + "message": "在 Windows 集群上启动 Kubernetes", + "description": "The label for category Launching Kubernetes on Windows Clusters in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Use Windows Clusters": { "message": "使用 Windows 集群", "description": "The label for category Use Windows Clusters in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Set Up Cloud Providers": { + "sidebar.tutorialSidebar.category.Setting up Cloud Providers": { "message": "设置 Cloud Provider", - "description": "The label for category Set Up Cloud Providers in sidebar tutorialSidebar" + "description": "The label for category Setting up Cloud Providers in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Launch Kubernetes with Rancher": { + "sidebar.tutorialSidebar.category.Launching Kubernetes with Rancher": { "message": "使用 Rancher 启动 Kubernetes", "description": "The label for category Launch Kubernetes with Rancher in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Use New Nodes in an Infra Provider": { + "sidebar.tutorialSidebar.category.Migrate to an Out-of-tree Cloud Provider": { + "message": "迁移到树外云提供商", + "description": "The label for category Migrate to an Out-of-tree Cloud Provider in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Launching New Nodes in an Infra Provider": { "message": "在基础设施提供商中使用新节点", - "description": "The label for category Use New Nodes in an Infra Provider in sidebar tutorialSidebar" + "description": "The label for category Launching New Nodes in an Infra Provider in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.vSphere": { - "message": "vSphere", - "description": "The label for category vSphere in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Creating a vSphere Cluster": { + "message": "创建 vSphere 集群", + "description": "The label for category Creating a vSphere Cluster in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Nutanix": { - "message": "Nutanix", - "description": "The label for category Nutanix in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Creating a Nutanix AOS Cluster": { + "message": "创建 Nutanix AOS 集群", + "description": "The label for category Creating a Nutanix AOS Cluster in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Kubernetes Resources Setup": { "message": "Kubernetes 资源设置", @@ -175,9 +183,9 @@ "message": "负载均衡和 Ingress Controller", "description": "The label for category Load Balancer and Ingress Controller in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Helm Charts in Rancher": { - "message": "Rancher 中的 Helm Chart", - "description": "The label for category Helm Charts in Rancher in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Helm Charts and Apps": { + "message": "Helm Charts 和 Apps", + "description": "The label for category Helm Charts and Apps in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Deploy Apps Across Clusters": { "message": "跨集群部署应用", @@ -191,13 +199,13 @@ "message": "高级用户指南", "description": "The label for category Advanced User Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Manage Projects": { - "message": "管理项目", - "description": "The label for category Manage Projects in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Project Administration": { + "message": "项目管理", + "description": "The label for category Project Administration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Manage Project Resource Quotas": { - "message": "管理项目资源配额", - "description": "The label for category Manage Project Resource Quotas in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Project Resource Quotas": { + "message": "项目资源配额", + "description": "The label for category Project Resource Quotas in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Monitoring/Alerting Guides": { "message": "Monitoring/Alerting 指南", @@ -207,33 +215,33 @@ "message": "Prometheus Federator 指南", "description": "The label for category Prometheus Federator Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Monitoring V2 Configuration Guides": { - "message": "Monitoring V2 配置指南", - "description": "The label for category Monitoring V2 Configuration Guides in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Monitoring Configuration Guides": { + "message": "Monitoring 配置指南", + "description": "The label for category Monitoring Configuration Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Advanced Configuration": { "message": "高级配置", "description": "The label for category Advanced Configuration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Istio Setup Guide": { + "sidebar.tutorialSidebar.category.Istio Setup Guides": { "message": "Istio 设置指南", - "description": "The label for category Istio Setup Guide in sidebar tutorialSidebar" + "description": "The label for category Istio Setup Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.CIS Scan Guides": { "message": "CIS 扫描指南", "description": "The label for category CIS Scan Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Enable Experimental Features": { + "sidebar.tutorialSidebar.category.Enabling Experimental Features": { "message": "启用实验功能", - "description": "The label for category Enable Experimental Features in sidebar tutorialSidebar" + "description": "The label for category Enabling Experimental Features in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Reference Guides": { "message": "参考指南", "description": "The label for category Reference Guides in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Best Practices": { + "sidebar.tutorialSidebar.category.Best Practice Guides": { "message": "最佳实践", - "description": "The label for category Best Practices in sidebar tutorialSidebar" + "description": "The label for category Best Practice Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Rancher Server": { "message": "Rancher Server", @@ -255,9 +263,9 @@ "message": "Rancher Server 配置", "description": "The label for category Rancher Server Configuration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.GKE Cluster Configuration": { + "sidebar.tutorialSidebar.category.GKE Cluster Configuration Reference": { "message": "GKE 集群配置", - "description": "The label for category GKE Cluster Configuration in sidebar tutorialSidebar" + "description": "The label for category GKE Cluster Configuration Reference in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Use Existing Nodes": { "message": "使用现有节点", @@ -283,9 +291,9 @@ "message": "备份和恢复配置", "description": "The label for category Backup & Restore Configuration in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Monitoring V2 Configuration": { - "message": "Monitoring V2 配置", - "description": "The label for category Monitoring V2 Configuration in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Monitoring Configuration Reference": { + "message": "Monitoring 配置", + "description": "The label for category Monitoring Configuration Reference in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Prometheus Federator": { "message": "Prometheus Federator", @@ -303,9 +311,9 @@ "message": "关于 API", "description": "The label for category About the API in sidebar tutorialSidebar" }, - "sidebar.tutorialSidebar.category.Rancher Security": { - "message": "Rancher 安全", - "description": "The label for category Rancher Security in sidebar tutorialSidebar" + "sidebar.tutorialSidebar.category.Rancher Security Guides": { + "message": "Rancher 安全指南", + "description": "The label for category Rancher Security Guides in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Hardening Guides": { "message": "加固指南", @@ -382,5 +390,25 @@ "sidebar.tutorialSidebar.category.Other Troubleshooting Tips": { "message": "其他故障排除提示", "description": "The label for category Other Troubleshooting Tips in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Example Workflows": { + "message": "工作流程示例", + "description": "The label for category Example Workflows in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Virtualization on Kubernetes with Harvester": { + "message": "使用 Harvester 在 Kubernetes 上进行虚拟化", + "description": "The label for category Virtualization on Kubernetes with Harvester in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Cloud Native Storage with Longhorn": { + "message": "使用 Longhorn 进行云原生存储", + "description": "The label for category Cloud Native Storage with Longhorn in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Container Security with Neuvector": { + "message": "使用 NeuVector 实现容器安全", + "description": "The label for category Container Security with Neuvector in sidebar tutorialSidebar" + }, + "sidebar.tutorialSidebar.category.Cluster API (CAPI) with Rancher Turtles": { + "message": "Cluster API (CAPI) 与 Rancher Turtles", + "description": "The label for category Cluster API (CAPI) with Rancher Turtles in sidebar tutorialSidebar" } } diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/faq/deprecated-features.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/faq/deprecated-features.md new file mode 100644 index 00000000000..16435260cfa --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/faq/deprecated-features.md @@ -0,0 +1,35 @@ +--- +title: Rancher 中已弃用的功能 +--- + + + + + +### Rancher 的弃用策略是什么？ + +我们已经在支持的[服务条款](https://rancher.com/support-maintenance-terms)中发布了官方的弃用策略。 + +### 在哪里可以了解 Rancher 中已弃用哪些功能？ + +Rancher 将在 GitHub 上发布的 Rancher 的[发版说明](https://github.com/rancher/rancher/releases)中发布已弃用的功能。有关已弃用的功能，请参阅以下的补丁版本： + +| Patch Version | Release Date | +| ----------------------------------------------------------------- | ------------------- | +| [2.7.12](https://github.com/rancher/rancher/releases/tag/v2.7.12) | 2024 年 3 月 28 日 | +| [2.7.11](https://github.com/rancher/rancher/releases/tag/v2.7.11) | 2024 年 3 月 1 日 | +| [2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) | 2024 年 2 月 8 日 | +| [2.7.9](https://github.com/rancher/rancher/releases/tag/v2.7.9) | 2023 年 10 月 26 日 | +| [2.7.8](https://github.com/rancher/rancher/releases/tag/v2.7.8) | 2023 年 10 月 5 日 | +| [2.7.7](https://github.com/rancher/rancher/releases/tag/v2.7.7) | 2023 年 9 月 28 日 | +| [2.7.6](https://github.com/rancher/rancher/releases/tag/v2.7.6) | 2023 年 8 月 30 日 | +| [2.7.5](https://github.com/rancher/rancher/releases/tag/v2.7.5) | 2023 年 6 月 29 日 | +| [2.7.4](https://github.com/rancher/rancher/releases/tag/v2.7.4) | 2023 年 5 月 31 日 | +| [2.7.3](https://github.com/rancher/rancher/releases/tag/v2.7.3) | 2023 年 4 月 24 日 | +| [2.7.2](https://github.com/rancher/rancher/releases/tag/v2.7.2) | 2023 年 4 月 11 日 | +| [2.7.1](https://github.com/rancher/rancher/releases/tag/v2.7.1) | 2023 年 1 月 24 日 | +| [2.7.0](https://github.com/rancher/rancher/releases/tag/v2.7.0) | 2022 年 11 月 16 日 | + +### 当一个功能被标记为弃用我可以得到什么样的预期？ + +当功能被标记为“已弃用”时，它依然可用并得到支持，允许按照常规的流程进行升级。一旦升级完成，用户/管理员应开始计划在升级到标记为已移除的版本之前放弃使用已弃用的功能。对于新的部署，建议不要使用已弃用的功能。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/faq/general-faq.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/faq/general-faq.md index 45d4ae835ae..5cf116534af 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/faq/general-faq.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/faq/general-faq.md @@ -2,68 +2,50 @@ title: 一般常见问题解答 --- + + + + 本文包含了用户常见的 Rancher 2.x 问题。 有关常见技术问题，请参阅[常见技术问题解答](technical-items.md)。 -
- -**Rancher 2.x 支持 Docker Swarm 和 Mesos 作为环境类型吗？** +## Rancher 2.x 支持 Docker Swarm 和 Mesos 作为环境类型吗？ 如果你在 Rancher 2.x 中创建环境，Swarm 和 Mesos 将不再是可选的标准选项。但是，Swarm 和 Mesos 还能继续作为可以部署的商店应用程序。这是一个艰难的决定，但这是大势所趋。比如说，15,000 多个集群可能只有大约 200 个在运行 Swarm。 -
+## 是否可以使用 Rancher 2.x 管理 Azure Kubernetes 服务？ -**是否可以使用 Rancher 2.x 管理 Azure Kubernetes 服务？** +是的。请参阅我们的[集群管理]((../how-to-guides/new-user-guides/manage-clusters/manage-clusters.md))指南，了解 AKS 上可用的 Rancher 功能，以及相关的 [AKS 的文档](../getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md) -是的。 +## Rancher 是否支持 Windows？ -
+Rancher 支持 Windows Server 1809 容器。有关如何使用 Windows Worker 节点设置集群的详细信息，请参阅[为 Windows 配置自定义集群](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)。 -**Rancher 是否支持 Windows？** - -Rancher 支持 Windows Server 1809 容器。有关如何使用 Windows Worker 节点设置集群的详细信息，请参阅[为 Windows 配置自定义集群](../pages-for-subheaders/use-windows-clusters.md)。 - -
- -**Rancher 是否支持 Istio？** +## Rancher 是否支持 Istio？ Rancher 支持 [Istio](../pages-for-subheaders/istio.md)。 -此外，Istio 是在我们的微型 PaaS “Rio” 中实现的，它可以运行在 Rancher 2.x 以及任何符合 CNCF 的 Kubernetes 集群上。详情请参阅[这里](https://rio.io/)。 - -
- -**Rancher 2.x 是否支持使用 Hashicorp 的 Vault 来存储密文？** +## Rancher 2.x 是否支持使用 Hashicorp 的 Vault 来存储密文？ 密文管理已在我们的 roadmap 上，但我们尚未将该功能分配给特定版本。 -
- -**Rancher 2.x 是否也支持 RKT 容器？** +## Rancher 2.x 是否也支持 RKT 容器？ 目前，我们只支持 Docker。 -
- -**Rancher 2.x 是否支持将 Calico、Contiv、Contrail、Flannel、Weave net 等网络插件用于嵌入和已注册的 Kubernetes？** +## Rancher 2.x 是否支持将 Calico、Contiv、Contrail、Flannel、Weave net 等网络插件用于嵌入和已注册的 Kubernetes？ Rancher 开箱即用地为 Kubernetes 集群提供了几个 CNI 网络插件，分别是 Canal、Flannel、Calico 和 Weave。有关官方支持的详细信息，请参阅 [Rancher 支持矩阵](https://rancher.com/support-maintenance-terms/)。 -
- -**Rancher 是否计划支持 Traefik？** +## Rancher 是否计划支持 Traefik？ 目前，我们不打算提供嵌入式 Traefik 支持，但我们仍在探索负载均衡方案。 -
- -**我可以将 OpenShift Kubernetes 集群导入 2.x 吗？** +## 我可以将 OpenShift Kubernetes 集群导入 2.x 吗？ 我们的目标是运行任何上游 Kubernetes 集群。因此，Rancher 2.x 应该可以与 OpenShift 一起使用，但我们尚未对此进行测试。 -
- -**Rancher 会集成 Longhorn 吗？** +## Rancher 会集成 Longhorn 吗？ 是的。Longhorn 已集成到 Rancher 2.5+ 中。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades.md index dee4c863bcf..5a7cd098a2d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/air-gapped-upgrades.md @@ -1,7 +1,11 @@ --- -title: 在离线环境中渲染 Helm 模板 +title: 在离线环境中升级 --- + + + + :::note 以下说明假设你已经按照[本页](upgrades.md)的 Kubernetes 升级说明操作（包括先决条件）到步骤 3：升级 Rancher。 @@ -14,13 +18,12 @@ title: 在离线环境中渲染 Helm 模板 根据你在安装过程中做出的选择，完成以下步骤之一。 -| 占位符 | 描述 | -------------|------------- -| `` | 输出压缩包的版本号。 | -| `` | 指向负载均衡器的 DNS 名称。 | -| `` | 你的私有镜像仓库的 DNS 名称。 | -| `` | 在 K8s 集群上运行的 cert-manager 版本。 | - +| 占位符 | 描述 | +| -------------------------------- | --------------------------------------- | +| `` | 输出压缩包的版本号。 | +| `` | 指向负载均衡器的 DNS 名称。 | +| `` | 你的私有镜像仓库的 DNS 名称。 | +| `` | 在 K8s 集群上运行的 cert-manager 版本。 | ### 选项 A：使用默认的自签名证书 @@ -35,6 +38,27 @@ helm template rancher ./rancher-.tgz --output-dir . \ --set useBundledSystemChart=true # Use the packaged Rancher system charts ``` +#### 解决 UPGRADE FAILED 错误 + +如果你遇到错误消息 `Error: UPGRADE FAILED: "rancher" has no deployed releases`，Rancher 可能是通过 `helm template` 命令安装的。要成功升级 Rancher，请改用以下命令： + +``` +helm template rancher ./rancher-.tgz --output-dir . \ + --no-hooks \ # prevent files for Helm hooks from being generated + --namespace cattle-system \ + --set hostname= \ + --set certmanager.version= \ + --set rancherImage=/rancher/rancher \ + --set systemDefaultRegistry= \ # Set a default private registry to be used in Rancher + --set useBundledSystemChart=true # Use the packaged Rancher system charts +``` + +执行 Helm 命令后，需要应用渲染后的模板： + +``` +kubectl -n cattle-system apply -R -f ./rancher +``` + ### 选项 B：使用 Kubernetes 密文从文件中获取证书 ```plain @@ -62,16 +86,6 @@ helm template rancher ./rancher-.tgz --output-dir . \ --set useBundledSystemChart=true # Use the packaged Rancher system charts ``` -### 应用已渲染的模板 - -将渲染的 manifest 目录复制到可以访问 Rancher Server 集群的系统中，并应用渲染的模板。 - -使用 `kubectl` 来应用渲染的 manifest。 - -```plain -kubectl -n cattle-system apply -R -f ./rancher -``` - ## 验证升级 登录 Rancher 以确认升级成功。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md new file mode 100644 index 00000000000..7b59a5b0db5 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md @@ -0,0 +1,352 @@ +--- +title: 在 Kubernetes 集群上安装/升级 Rancher +description: 了解如何在开发和生产环境中安装 Rancher。了解单节点和高可用安装 +--- + + + + + +在本节中，你将学习如何使用 Helm CLI 在 Kubernetes 集群上部署 Rancher。 + +## 先决条件 + +- [Kubernetes 集群](#kubernetes-集群) +- [Ingress Controller](#ingress-controller) +- [CLI 工具](#cli-工具) + +### Kubernetes 集群 + +设置 Rancher Server 的本地 Kubernetes 集群。 + +Rancher 可以安装在任何 Kubernetes 集群上。这个集群可以使用上游 Kubernetes，也可以使用 Rancher 的 Kubernetes 发行版之一，也可以是来自 Amazon EKS 等提供商的托管 Kubernetes 集群。 + +你可参考以下教程，以获得设置 Kubernetes 集群的帮助： + +- **RKE**：[安装 RKE Kubernetes 集群的教程](../../../how-to-guides/new-user-guides/kubernetes-cluster-setup/rke1-for-rancher.md)；[为高可用 RKE 集群设置基础设施的教程](../../../how-to-guides/new-user-guides/infrastructure-setup/ha-rke1-kubernetes-cluster.md)。 +- **K3s**：[安装 K3s Kubernetes 集群的教程](../../../how-to-guides/new-user-guides/kubernetes-cluster-setup/k3s-for-rancher.md)；[设置高可用 K3s 集群的基础设施的教程](../../../how-to-guides/new-user-guides/infrastructure-setup/ha-k3s-kubernetes-cluster.md)。 +- **RKE2:** ：[安装 RKE2 Kubernetes 集群的教程](../../../how-to-guides/new-user-guides/kubernetes-cluster-setup/rke2-for-rancher.md)；[设置高可用 RKE2 集群的基础设施的教程](../../../how-to-guides/new-user-guides/infrastructure-setup/ha-rke2-kubernetes-cluster.md)。 +- **Amazon EKS**：[在 Amazon EKS 上安装 Rancher 以及如何安装 Ingress Controller 以访问 Rancher Server](rancher-on-amazon-eks.md)。 +- **AKS**：[使用 Azure Kubernetes 服务安装 Rancher 以及如何安装 Ingress Controller 以访问 Rancher Server](rancher-on-aks.md)。 +- **GKE**：有关如何使用 GKE 安装 Rancher，包括如何安装 Ingress Controller 以便可以访问 Rancher Server，请参阅[此页面](rancher-on-gke.md)。GKE 创建 Kubernetes 集群时有两种运行模式，分别是 Autopilot 和 Standard 模式。Autopilot 模式的集群配置对编辑 kube-system 命名空间有限制。但是，Rancher 在安装时需要在 kube-system 命名空间中创建资源。因此，你将无法在以 Autopilot 模式创建的 GKE 集群上安装 Rancher。 + +### Ingress Controller + +Rancher UI 和 API 通过 Ingress 公开。换言之，安装 Rancher 的 Kubernetes 集群必须包含一个 Ingress Controller。 + +对于 RKE、RKE2 和 K3s，你不需要手动安装 Ingress Controller，因为它是默认安装的。 + +对于默认不包含 Ingress Controller 的发行版（例如 EKS、GKE 或 AKS 等托管 Kubernetes 集群），你必须先部署 Ingress Controller。请注意，Rancher Helm Chart 默认情况下不会在 Ingress 上设置 `ingressClassName`。因此，你必须将 Ingress Controller 配置为在没有 `ingressClassName` 的情况下也可以监视 Ingress。 + +上面的 **Amazon EKS**、**AKS** 和 **GKE** 教程中包含了示例。 + +### CLI 工具 + +设置 Kubernetes 集群需要以下 CLI 工具。请确保这些工具已安装并在你的 `$PATH` 中可用。 + +- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/#install-kubectl) - Kubernetes 命令行工具。 +- [Helm](https://docs.helm.sh/using_helm/#installing-helm) - Kubernetes 的包管理器。请参见 [Helm 版本要求](../resources/helm-version-requirements.md)选择 Helm 版本来安装 Rancher。请为你的具体平台参见 [Helm 项目提供的说明](https://helm.sh/docs/intro/install/)。 + +## 安装 Rancher Helm Chart + +Rancher 是使用 Kubernetes 的 [Helm](https://helm.sh/) 包管理器安装的。Helm Chart 为 Kubernetes YAML 清单文件提供了模板语法。通过 Helm，用户可以创建可配置的 deployment，而不仅仅只能使用静态文件。 + +如果系统无法直接访问互联网，请参见[离线环境：Kubernetes 安装](../other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha.md)。 + +如果要指定安装的 Rancher 版本，请参见[选择 Rancher 版本](../resources/choose-a-rancher-version.md)。 + +如果要指定用于安装 Rancher 的 Helm 版本，请参见[Helm 版本要求](../resources/helm-version-requirements.md)。 + +:::note + +本安装指南假定你使用的是 Helm 3。 + +::: + +要设置 Rancher： + +1. [添加 Helm Chart 仓库](#1-添加-helm-chart-仓库) +2. [为 Rancher 创建命名空间](#2-为-rancher-创建命名空间) +3. [选择 SSL 配置](#3-选择-ssl-配置) +4. [安装 cert-manager](#4-安装-cert-manager)（除非你自带证书，否则 TLS 将在负载均衡器上终止） +5. [使用 Helm 和你选择的证书选项安装 Rancher](#5-根据你选择的证书选项通过-helm-安装-rancher) +6. [验证 Rancher Server 是否部署成功](#6-验证-rancher-server-是否部署成功) +7. [保存选项](#7-保存选项) + +### 1. 添加 Helm Chart 仓库 + +执行 `helm repo add` 命令，以添加包含安装 Rancher 的 Chart 的 Helm Chart 仓库。有关如何选择仓库，以及哪个仓库最适合你的用例，请参见[选择 Rancher 版本](../getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md)。 + +- Latest：建议用于试用最新功能 + ``` + helm repo add rancher-latest https://releases.rancher.com/server-charts/latest + ``` +- Stable：建议用于生产环境 + ``` + helm repo add rancher-stable https://releases.rancher.com/server-charts/stable + ``` +- Alpha：即将发布的实验性预览。 + ``` + helm repo add rancher-alpha https://releases.rancher.com/server-charts/alpha + ``` + 注意：不支持升级到 Alpha 版、从 Alpha 版升级或在 Alpha 版之间升级。 + +### 2. 为 Rancher 创建命名空间 + +你需要定义一个 Kubernetes 命名空间，用于安装由 Chart 创建的资源。这个命名空间的名称为 `cattle-system`： + +``` +kubectl create namespace cattle-system +``` + +### 3. 选择 SSL 配置 + +Rancher Management Server 默认需要 SSL/TLS 配置来保证访问的安全性。 + +:::note + +如果你想在外部终止 SSL/TLS，请参见[外部负载均衡器的 TLS 终止](../getting-started/installation-and-upgrade/installation-references/helm-chart-options.md#外部-tls-终止)。 + +::: + +你可以从以下三种证书来源中选择一种，用于在 Rancher Server 中终止 TLS： + +- **Rancher 生成的 TLS 证书**：要求你在集群中安装 `cert-manager`。Rancher 使用 `cert-manager` 签发并维护证书。Rancher 会生成自己的 CA 证书，并使用该 CA 签署证书。然后 `cert-manager`负责管理该证书。 +- **Let's Encrypt**：Let's Encrypt 选项也需要使用 `cert-manager`。但是，在这种情况下，cert-manager 与 Let's Encrypt 的特殊颁发者相结合，该颁发者执行获取 Let's Encrypt 颁发的证书所需的所有操作（包括请求和验证）。此配置使用 HTTP 验证（`HTTP-01`），因此负载均衡器必须具有可以从互联网访问的公共 DNS 记录。 +- **你已有的证书**：使用已有的 CA 颁发的公有或私有证书。Rancher 将使用该证书来保护 WebSocket 和 HTTPS 流量。在这种情况下，你必须上传名称分别为 `tls.crt` 和 `tls.key`的 PEM 格式的证书以及相关的密钥。如果你使用私有 CA，则还必须上传该 CA 证书。这是由于你的节点可能不信任此私有 CA。Rancher 将获取该 CA 证书，并从中生成一个校验和，各种 Rancher 组件将使用该校验和来验证其与 Rancher 的连接。 + +| 配置 | Helm Chart 选项 | 是否需要 cert-manager | +| -------------------------- | -------------------------------- | -------------------------- | +| Rancher 生成的证书（默认） | `ingress.tls.source=rancher` | [是](#4-安装-cert-manager) | +| Let’s Encrypt | `ingress.tls.source=letsEncrypt` | [是](#4-安装-cert-manager) | +| 你已有的证书 | `ingress.tls.source=secret` | 否 | + +### 4. 安装 cert-manager + +> 如果你使用自己的证书文件（`ingress.tls.source=secret`）或使用[外部负载均衡器的 TLS 终止](../installation-references/helm-chart-options.md#外部-tls-终止)，你可以跳过此步骤。 + +仅在使用 Rancher 生成的证书（`ingress.tls.source=rancher`）或 Let's Encrypt 颁发的证书（`ingress.tls.source=letsEncrypt`）时，才需要安装 cert-manager。 + +
+ 单击展开 + +:::note 重要提示： + +由于 cert-manager 的最新改动，你需要升级 cert-manager 版本。如果你需要升级 Rancher 并使用低于 0.11.0 的 cert-manager 版本，请参见[升级文档](../resources/upgrade-cert-manager.md)。 + +::: + +这些说明来自 [cert-manager 官方文档](https://cert-manager.io/docs/installation/kubernetes/#installing-with-helm)。 + +:::note + +要查看自定义 cert-manager 安装的选项（包括集群使用 PodSecurityPolicies 的情况），请参阅 [cert-manager 文档](https://artifacthub.io/packages/helm/cert-manager/cert-manager#configuration)。 + +::: + +``` +# 如果你手动安装了CRD，而不是在 Helm 安装命令中添加了 `--set installCRDs=true` 选项，你应该在升级 Helm Chart 之前升级 CRD 资源。 +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download//cert-manager.crds.yaml + +# 添加 Jetstack Helm 仓库 +helm repo add jetstack https://charts.jetstack.io + +# 更新本地 Helm Chart 仓库缓存 +helm repo update + +# 安装 cert-manager Helm Chart +helm install cert-manager jetstack/cert-manager \ + --namespace cert-manager \ + --create-namespace +``` + +安装完 cert-manager 后，你可以通过检查 cert-manager 命名空间中正在运行的 Pod 来验证它是否已正确部署： + +``` +kubectl get pods --namespace cert-manager + +NAME READY STATUS RESTARTS AGE +cert-manager-5c6866597-zw7kh 1/1 Running 0 2m +cert-manager-cainjector-577f6d9fd7-tr77l 1/1 Running 0 2m +cert-manager-webhook-787858fcdb-nlzsq 1/1 Running 0 2m +``` + +
+ +### 5. 根据你选择的证书选项，通过 Helm 安装 Rancher + +不同的证书配置需要使用不同的 Rancher 安装命令。 + +但是，无论证书如何配置，Rancher 在 `cattle-system` 命名空间中的安装名称应该总是 `rancher`。 + +:::tip 测试和开发： + +这个安装 Rancher 的最终命令需要一个将流量转发到 Rancher 的域名。如果你使用 Helm CLI 设置概念证明，则可以在传入 `hostname` 选项时使用伪域名。伪域名的一个例子是 `.sslip.io`，这会把 Rancher 暴露在它运行的 IP 上。生产安装中要求填写真实的域名。 + +::: + + + + +默认情况是使用 Rancher 生成 CA，并使用 `cert-manager` 颁发用于访问 Rancher Server 接口的证书。 + +由于 `rancher` 是 `ingress.tls.source` 的默认选项，因此在执行 `helm install` 命令时，我们不需要指定 `ingress.tls.source`。 + +- 将 `hostname` 设置为解析到你的负载均衡器的 DNS 名称。 +- 将 `bootstrapPassword` 设置为 `admin` 用户独有的值。 +- 如果你需要安装指定的 Rancher 版本，使用 `--version` 标志，例如 `--version 2.7.0`。 +- 对于 Kubernetes v1.25 或更高版本，使用 Rancher v2.7.2-v2.7.4 时，将 `global.cattle.psp.enabled` 设置为 `false`。对于 Rancher v2.7.5 及更高版本来说，这不是必需的，但你仍然可以手动设置该选项。 + +``` +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set bootstrapPassword=admin +``` + +如果你安装的是 alpha 版本，Helm 会要求你在安装命令中添加 `--devel` 选项： + +``` +helm install rancher rancher-alpha/rancher --devel +``` + +等待 Rancher 运行： + +``` +kubectl -n cattle-system rollout status deploy/rancher +Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... +deployment "rancher" successfully rolled out +``` + + + + +此选项使用 `cert-manager` 来自动请求和续订 [Let's Encrypt](https://letsencrypt.org/) 证书。Let's Encrypt 是免费的，而且是受信的 CA，因此可以为你提供有效的证书。 + +:::note + +由于 HTTP-01 质询只能在端口 80 上完成，因此你需要打开端口 80。 + +::: + +在以下命令中， + +- 将 `hostname` 设置为公有 DNS 记录。 +- 将 `bootstrapPassword` 设置为 `admin` 用户独有的值。 +- 将 `ingress.tls.source` 设置为 `letsEncrypt`。 +- 将 `letsEncrypt.email` 设置为可通讯的电子邮件地址，用于发送通知（例如证书到期的通知）。 +- 将 `letsEncrypt.ingress.class` 设为你的 Ingress Controller（例如 `traefik`，`nginx`，`haproxy`） +- 对于 Kubernetes v1.25 或更高版本，使用 Rancher v2.7.2-v2.7.4 时，将 `global.cattle.psp.enabled` 设置为 `false`。对于 Rancher v2.7.5 及更高版本来说，这不是必需的，但你仍然可以手动设置该选项。 + +``` +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set bootstrapPassword=admin \ + --set ingress.tls.source=letsEncrypt \ + --set letsEncrypt.email=me@example.org \ + --set letsEncrypt.ingress.class=nginx +``` + +如果你安装的是 alpha 版本，Helm 会要求你在安装命令中添加 `--devel` 选项： + +``` +helm install rancher rancher-alpha/rancher --devel +``` + +等待 Rancher 运行： + +``` +kubectl -n cattle-system rollout status deploy/rancher +Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... +deployment "rancher" successfully rolled out +``` + + + +在此选项中，你使用你自己的证书来创建 Kubernetes 密文，以供 Rancher 使用。 + +运行这个命令时，`hostname` 选项必须与服务器证书中的 `Common Name` 或 `Subject Alternative Names` 条目匹配，否则 Ingress controller 将无法正确配置。 + +虽然技术上仅需要 `Subject Alternative Names` 中有一个条目，但是拥有一个匹配的 `Common Name` 可以最大程度地提高与旧版浏览器/应用的兼容性。 + +:::note + +如果你想检查证书是否正确，请查看[如何在服务器证书中检查 Common Name 和 Subject Alternative Names](../../../faq/technical-items.md#如何在服务器证书中检查-common-name-和-subject-alternative-names)。 + +::: + +- 设置 `hostname`。 +- 将 `bootstrapPassword` 设置为 `admin` 用户独有的值。 +- 将 `ingress.tls.source` 设置为 `secret`。 +- 对于 Kubernetes v1.25 或更高版本，使用 Rancher v2.7.2-v2.7.4 时，将 `global.cattle.psp.enabled` 设置为 `false`。对于 Rancher v2.7.5 及更高版本来说，这不是必需的，但你仍然可以手动设置该选项。 + +``` +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set bootstrapPassword=admin \ + --set ingress.tls.source=secret +``` + +如果你安装的是 alpha 版本，Helm 会要求你在安装命令中添加 `--devel` 选项： + +``` +helm install rancher rancher-alpha/rancher --devel +``` + +如果你使用的是私有 CA 证书，请在命令中增加 `--set privateCA=true`。 + +``` +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set bootstrapPassword=admin \ + --set ingress.tls.source=secret \ + --set privateCA=true +``` + +**添加 TLS 密文（千万不要遗漏此步骤）**：现在 Rancher 已经完成部署，你还需要参考[添加 TLS 密文](../resources/add-tls-secrets.md)发布证书文件，以便 Rancher 和 Ingress Controller 可以使用它们。 + + + + +Rancher Chart 有许多选项，用于为你的具体环境自定义安装。以下是一些常见的高级方案： + +- [HTTP 代理](../installation-references/helm-chart-options.md#http-代理) +- [私有容器镜像仓库](../installation-references/helm-chart-options.md#私有仓库和离线安装) +- [外部负载均衡器上的 TLS 终止](../installation-references/helm-chart-options.md#外部-tls-终止) + +如需获取完整的选项列表，请参见 [Chart 选项](../installation-references/helm-chart-options.md)。 + +### 6. 验证 Rancher Server 是否部署成功 + +添加密文后，检查 Rancher 是否已成功运行： + +``` +kubectl -n cattle-system rollout status deploy/rancher +Waiting for deployment "rancher" rollout to finish: 0 of 3 updated replicas are available... +deployment "rancher" successfully rolled out +``` + +如果你看到 `error: deployment "rancher" exceeded its progress deadline` 这个错误，可运行以下命令来检查 deployment 的状态： + +``` +kubectl -n cattle-system get deploy rancher +NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE +rancher 3 3 3 3 3m +``` + +`DESIRED` 和 `AVAILABLE`的个数应该相同。 + +### 7. 保存选项 + +请保存你使用的 `--set` 选项。使用 Helm 升级 Rancher 到新版本时，你将需要使用相同的选项。 + +### 安装完成 + +安装已完成。现在 Rancher Server 应该已经可以正常运行了。 + +使用浏览器打开把流量转发到你的负载均衡器的 DNS 域名。然后，你就会看到一个漂亮的登录页面了。 + +如果遇到任何问题，请参见[故障排除](troubleshooting.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md index 22de9e7af35..beb80e5f515 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md @@ -2,11 +2,15 @@ title: 在 Azure Kubernetes Service 上安装 Rancher --- + + + + 本文介绍了如何在微软的 Azure Kubernetes Service (AKS) 上安装 Rancher。 本指南使用命令行工具来配置一个带有 Ingress 的 AKS 集群。如果你更喜欢使用 Azure 门户来配置集群，请参见[官方文档](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-portal)。 -如果你已有一个 AKS Kubernetes 集群，请直接跳到[安装 Ingress](#5-安装-ingress) 的步骤，然后按照[此页](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。 +如果你已有一个 AKS Kubernetes 集群，请直接跳到[安装 Ingress](#5-安装-ingress) 的步骤，然后按照[此页](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。 ## 先决条件 @@ -19,7 +23,7 @@ title: 在 Azure Kubernetes Service 上安装 Rancher - [Microsoft Azure 账号](https://azure.microsoft.com/en-us/free/)：用于创建部署 Rancher 和 Kubernetes 的资源。 - [Microsoft Azure 订阅](https://docs.microsoft.com/en-us/azure/cost-management-billing/manage/create-subscription#create-a-subscription-in-the-azure-portal)：如果你没有的话，请访问此链接查看如何创建 Microsoft Azure 订阅。 - [Micsoroft Azure 租户](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-create-new-tenant)：访问此链接并参考教程以创建 Microsoft Azure 租户。 -- 你的订阅有足够的配额，至少有 2 个 vCPU。有关 Rancher Server 资源要求的详情，请参见[此节](../../../pages-for-subheaders/installation-requirements.md#rke-和托管-kubernetes)。 +- 你的订阅有足够的配额，至少有 2 个 vCPU。有关 Rancher Server 资源要求的详情，请参见[此节](../installation-requirements/installation-requirements.md)。 - 在 Azure 中用 Helm 安装 Rancher 时，请使用 L7 负载均衡器来避免网络问题。详情请参见 [Azure 负载均衡器限制](https://docs.microsoft.com/en-us/azure/load-balancer/components#limitations)。 ## 1. 准备你的工作站 @@ -134,7 +138,7 @@ ingress-nginx-controller LoadBalancer 10.0.116.18 40.31.180.83 80:31229 ## 8. 安装 Rancher Helm Chart -按照[本页](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。任何 Kubernetes 发行版上安装的 Rancher 的 Helm 说明都是一样的。 +按照[本页](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。任何 Kubernetes 发行版上安装的 Rancher 的 Helm 说明都是一样的。 安装 Rancher 时，使用上一步获取的 DNS 名称作为 Rancher Server 的 URL。它可以作为 Helm 选项传递进来。例如，如果 DNS 名称是 `rancher.my.org`，你需要使用 `--set hostname=rancher.my.org` 选项来运行 Helm 安装命令。 @@ -144,4 +148,4 @@ ingress-nginx-controller LoadBalancer 10.0.116.18 40.31.180.83 80:31229 --set ingress.ingressClassName=nginx ``` -请参阅[Helm 安装命令](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 +请参阅[Helm 安装命令](install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md index 335e3fee425..378c4bea60f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md @@ -2,9 +2,13 @@ title: 在 Amazon EKS 上安装 Rancher --- + + + + 本文介绍了如何在 Amazon EKS 集群上安装 Rancher。你也可以[通过 AWS Marketplace 安装 Rancher](../../quick-start-guides/deploy-rancher-manager/aws-marketplace.md)。 -如果你已经有一个 EKS Kubernetes 集群，请直接跳转到[安装 Ingress](#5-安装-ingress)这个步骤。然后按照[此处](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的步骤安装 Rancher Helm Chart。 +如果你已经有一个 EKS Kubernetes 集群，请直接跳转到[安装 Ingress](#5-安装-ingress)这个步骤。然后按照[此处](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的步骤安装 Rancher Helm Chart。 ## 为 Rancher Server 创建 EKS 集群 @@ -37,12 +41,12 @@ aws configure 输入以下参数： -| 值 | 描述 | -|-------|-------------| -| AWS Access Key ID | 具有 EKS 权限的 IAM 用户的访问密钥凭证。 | -| AWS Secret Access Key | 具有 EKS 权限的 IAM 用户的密文密钥凭证。 | -| Default region name | 集群节点所在的 [AWS 区域](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html#Concepts.RegionsAndAvailabilityZones.Regions)。 | -| Default output format | 输入 `json`。 | +| 值 | 描述 | +| --------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| AWS Access Key ID | 具有 EKS 权限的 IAM 用户的访问密钥凭证。 | +| AWS Secret Access Key | 具有 EKS 权限的 IAM 用户的密文密钥凭证。 | +| Default region name | 集群节点所在的 [AWS 区域](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html#Concepts.RegionsAndAvailabilityZones.Regions)。 | +| Default output format | 输入 `json`。 | ### 3. 创建 EKS 集群 @@ -138,7 +142,7 @@ ingress-nginx-controller LoadBalancer 10.100.90.18 a904a952c73bf4f668a17c4 ### 8. 安装 Rancher Helm Chart -按照[本页](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。任何 Kubernetes 发行版上安装的 Rancher 的 Helm 说明都是一样的。 +按照[本页](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的说明安装 Rancher Helm Chart。任何 Kubernetes 发行版上安装的 Rancher 的 Helm 说明都是一样的。 安装 Rancher 时，使用上一步获取的 DNS 名称作为 Rancher Server 的 URL。它可以作为 Helm 选项传递进来。例如，如果 DNS 名称是 `rancher.my.org`，你需要使用 `--set hostname=rancher.my.org` 选项来运行 Helm 安装命令。 @@ -148,4 +152,4 @@ ingress-nginx-controller LoadBalancer 10.100.90.18 a904a952c73bf4f668a17c4 --set ingress.ingressClassName=nginx ``` -请参阅[Helm 安装命令](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 +请参阅[Helm 安装命令](install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md index ac40b92087d..42e78833ccc 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md @@ -2,15 +2,19 @@ title: 在 GKE 集群上安装 Rancher --- + + + + 在本节中，你将学习如何使用 GKE 安装 Rancher。 -如果你已经有一个 GKE Kubernetes 集群，请直接跳转到[安装 Ingress](#7-安装-ingress)这个步骤。然后按照[此处](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的步骤安装 Rancher Helm Chart。 +如果你已经有一个 GKE Kubernetes 集群，请直接跳转到[安装 Ingress](#7-安装-ingress)这个步骤。然后按照[此处](install-upgrade-on-a-kubernetes-cluster.md#安装-rancher-helm-chart)的步骤安装 Rancher Helm Chart。 ## 先决条件 - 你需要有一个 Google 账号。 - 你需要有一个 Google Cloud Billing 账号。你可使用 Google Cloud Console 来管理你的 Cloud Billing 账号。有关 Cloud Console 的详情，请参见 [ Console 通用指南](https://support.google.com/cloud/answer/3465889?hl=en&ref_topic=3340599)。 -- 你需要至少一个在用的 IP 地址和至少 2 个 CPU 的云配额。有关 Rancher Server 的硬件要求，请参见[本节](../../../pages-for-subheaders/installation-requirements.md#rke-和托管-kubernetes)。 +- 你需要至少一个在用的 IP 地址和至少 2 个 CPU 的云配额。有关 Rancher Server 的硬件要求，请参见[本节](../installation-requirements/installation-requirements.md)。 ## 1. 启用 Kubernetes Engine API @@ -41,25 +45,28 @@ Cloud Shell 是一个 shell 环境，用于管理托管在 Google Cloud 上的 ``` gcloud components install kubectl ``` + 后面的步骤会配置 `kubectl`，使其用于使用新的 GKE 集群。 + 1. 如果 Helm 3 未安装的话，[安装 Helm 3](https://helm.sh/docs/intro/install/)。 1. 使用 `HELM_EXPERIMENTAL_OCI` 变量来启用 Helm 的实验功能 [OCI 镜像支持](https://github.com/helm/community/blob/master/hips/hip-0006.md)。把以下行添加到 `~/.bashrc` （或 macOS 中的 `~/.bash_profile`，或者你的 shell 存储环境变量的地方）： ``` export HELM_EXPERIMENTAL_OCI=1 ``` + 1. 运行以下命令来加载你更新的 `.bashrc` 文件： ``` source ~/.bashrc ``` + 如果你运行的是 macOS，使用这个命令: + ``` source ~/.bash_profile ``` - - ## 3. 配置 gcloud CLI 选择以下方法之一配置默认的 gcloud 设置： @@ -75,13 +82,14 @@ Cloud Shell 是一个 shell 环境，用于管理托管在 Google Cloud 上的 ``` gcloud init ``` + 如果你在远程服务器上使用 SSH，使用 --console-only 标志，以防止该命令启动浏览器。 ``` gcloud init --console-only ``` -2. 按照指示，以授权 gcloud 使用你的 Google Cloud 账户，并选择你创建的新项目。 +2. 按照指示，以授权 gcloud 使用你的 Google Cloud 账户，并选择你创建的新项目。 @@ -190,7 +198,7 @@ ingress-nginx-controller LoadBalancer 10.3.244.156 35.233.206.34 80:3187 --set ingress.ingressClassName=nginx ``` -请参阅[Helm 安装命令](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 +请参阅[Helm 安装命令](install-upgrade-on-a-kubernetes-cluster.md#5-根据你选择的证书选项通过-helm-安装-rancher)了解你的证书选项。 在 Rancher v2.7.5 中，如果你打算在集群上使用默认的 GKE Ingress 而不启用 VPC 原生的集群模式，则需要设置以下标志： diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md index 5570d3c4753..edbbff48f9f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md @@ -2,16 +2,41 @@ title: 回滚 --- -## 使用 Rancher 2.6.4+ 进行回滚的其他步骤 + + + + +本页概述了如何在升级后将 Rancher 回滚到以前的版本。 + +当出现以下情况时，请按照本页的说明进行操作： + +- 备份完成后，正在运行的 Rancher 实例已经升级到新版本。 +- 上游（本地）集群备份位置相同。 + +:::tip + +- 按照以下步骤[迁移 Rancher](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)。 +- 如果你需要将 Rancher 恢复到之前的状态，请参阅[回滚文档](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md). + +::: + +## 特殊场景的替代步骤 + +在以下场景下，需要执行回滚的替代步骤： + +- 从 v2.6.4 及更高版本回滚到 v2.6.x 的早期版本。 +- 从 v2.7.7 及更高版本回滚到 v2.7.x 的早期版本。 Rancher v2.6.4 将 cluster-api 模块从 v0.4.4 升级到 v1.0.2。反过来，cluster-api 的 v1.0.2 版本将集群 API 的自定义资源定义 (CRD) 从 `cluster.x-k8s.io/v1alpha4` 升级到 `cluster.x-k8s.io/v1beta1`。当你尝试将 Rancher v2.6.4 回滚到以前版本的 Rancher v2.6.x 时，CRD 升级到 v1beta1 会导致回滚失败。这是因为使用旧 apiVersion (v1alpha4) 的 CRD 与 v1beta1 不兼容。 -要避免回滚失败，你需要在尝试恢复操作或回滚**之前**运行以下 Rancher 脚本： +Rancher v2.7.7 中，`rancher-provisioning-capi` 应用作为嵌入式 cluster-api controllers 的替代品将会被自动安装到上游（本地）集群中。如果上游集群同时包含`rancher-provisioning-capi` 应用以及 Rancher v2.7.6 及更早版本的控制器，则会出现冲突和意外的错误。因此，当你尝试从 Rancher v2.7.7 迁移到 Rancher v2.7.x 的任何早期版本，都需要采取一些替代步骤。 -* `verify.sh`：检查集群中是否有任何与 Rancher 相关的资源。 -* `cleanup.sh`：清理集群。 +### 步骤一：清理上游（本地）集群 -有关详细信息和源代码，请参阅 [rancher/rancher-cleanup repo](https://github.com/rancher/rancher-cleanup)。 +要避免回滚失败，请按照以下[说明](https://github.com/rancher/rancher-cleanup/blob/main/README.md)，在你需要在尝试恢复操作或回滚**之前**运行以下脚本： + +- `cleanup.sh`：清理集群。 +- `verify.sh`：检查集群中是否有任何与 Rancher 相关的资源。 :::caution @@ -19,12 +44,20 @@ Rancher v2.6.4 将 cluster-api 模块从 v0.4.4 升级到 v1.0.2。反过来，c ::: -### 从 v2.6.4+ 回滚到较低版本的 v2.6.x +**结果：** 所有与 Rancher 相关的资源都会从上游（本地）集群中清理掉。 -1. 按照[说明](https://github.com/rancher/rancher-cleanup/blob/main/README.md)运行脚本。 -1. 按照[说明](https://rancher.com/docs/rancher/v2.6/en/backups/migrating-rancher/)在现有集群上安装 rancher-backup Helm Chart 并恢复之前的状态。 - 1. 省略步骤 3。 - 1. 执行到步骤 4 时，在要回滚到的 local 集群上安装 Rancher 2.6.x 版本。 +有关详细信息和源代码，请参阅 [rancher/rancher-cleanup repo](https://github.com/rancher/rancher-cleanup)。 + +### 步骤二：恢复备份并启动 Rancher + +此时，上游集群中不应该存在与 Rancher 相关的资源了。因此下一步与将 Rancher 迁移到不包含 Rancher 资源的新集群操作相同。 + +请参照以下[说明](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)安装 Rancher-Backup Helm chart 并将 Rancher 恢复到之前的状态。 + +请记住： + +1. 可以跳过步骤 3，因为如果之前安装了 cert-manager 应用，该程序应该仍存在于上游（本地）集群中。 +2. 在步骤 4，安装你想要回滚的 Rancher 版本。 ## 回滚到 Rancher 2.5.0+ @@ -36,20 +69,20 @@ Rancher v2.6.4 将 cluster-api 模块从 v0.4.4 升级到 v1.0.2。反过来，c :::note 重要提示： -* 请按照此页面上的说明在已备份的同一集群上还原 Rancher。要把 Rancher 迁移到新集群，请参照步骤[迁移 Rancher](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)。 +- 请按照此页面上的说明在已备份的同一集群上还原 Rancher。要把 Rancher 迁移到新集群，请参照步骤[迁移 Rancher](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)。 -* 在使用相同设置恢复 Rancher 时，Rancher deployment 在恢复开始前被手动缩减，然后 Operator 将在恢复完成后将其缩回。因此，在恢复完成之前，Rancher 和 UI 都将不可用。如果 UI 不可用时，你可使用 `kubectl create -f restore.yaml`YAML 恢复文件来使用初始的集群 kubeconfig。 +- 在使用相同设置恢复 Rancher 时，Rancher deployment 在恢复开始前被手动缩减，然后 Operator 将在恢复完成后将其缩回。因此，在恢复完成之前，Rancher 和 UI 都将不可用。如果 UI 不可用时，你可使用 `kubectl create -f restore.yaml`YAML 恢复文件来使用初始的集群 kubeconfig。 ::: -### 创建 Restore 自定义资源 +### 步骤一：创建 Restore 自定义资源 1. 点击 **☰ > 集群管理**。 1. 找到你的本地集群，并点击 **Explore**。 1. 在左侧导航栏中，点击 **Rancher 备份 > 还原**。 :::note - 如果 Rancher Backups 应用不可见，你需要到 **Apps** 的 Charts 页面中安装应用。详情请参见[此处](../../../pages-for-subheaders/helm-charts-in-rancher.md#charts)。 + 如果 Rancher Backups 应用不可见，你需要到 **Apps** 的 Charts 页面中安装应用。详情请参见[此处](../../../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md#访问-charts)。 ::: @@ -74,6 +107,7 @@ Rancher v2.6.4 将 cluster-api 模块从 v0.4.4 升级到 v1.0.2。反过来，c region: us-west-2 endpoint: s3.us-west-2.amazonaws.com ``` + 如需获得配置 Restore 的帮助，请参见[配置参考](../../../reference-guides/backup-restore-configuration/restore-configuration.md)和[示例](../../../reference-guides/backup-restore-configuration/examples.md)。 1. 单击**创建**。 @@ -91,7 +125,7 @@ kubectl get pods -n cattle-resources-system kubectl logs -n cattle-resources-system -f ``` -### 回滚到上一个 Rancher 版本 +### 步骤二：回滚到上一个 Rancher 版本 你可以使用 Helm CLI 回滚 Rancher。要回滚到上一个版本： @@ -111,7 +145,7 @@ helm history rancher -n cattle-system helm rollback rancher 3 -n cattle-system ``` -## 回滚到 Rancher 2.2-2.4 +## 回滚到 Rancher 2.2-2.4+ 要回滚到 2.5 之前的 Rancher 版本，参考此处的步骤[恢复备份 — Kubernetes 安装](/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher-launched-kubernetes-clusters-from-backup.md)。如果恢复 Rancher Server 的集群的某个快照，Rancher 的版本以及状态均会恢复回到快照时的版本和状态。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades.md index 711077b6635..7323429fda2 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades.md @@ -2,13 +2,16 @@ title: 升级 --- + + + + 本文介绍如何升级使用 Helm 安装在 Kubernetes 集群上的 Rancher Server。这些步骤也适用于使用 Helm 进行的离线安装。 有关使用 Docker 安装的 Rancher 的升级说明，请参见[本页。](../other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md) 如需升级 Kubernetes 集群中的组件，或 [Kubernetes services](https://rancher.com/docs/rke/latest/en/config-options/services/) 或 [附加组件（add-on）](https://rancher.com/docs/rke/latest/en/config-options/add-ons/)的定义，请参见 [RKE 升级文档](https://rancher.com/docs/rke/latest/en/upgrades/)的 Rancher Kubernetes 引擎。 - ## 先决条件 ### 访问 kubeconfig @@ -24,15 +27,18 @@ kubeconfig 也可以通过 `--kubeconfig` 标签（详情请参见 https://helm. 如需查看每个 Rancher 版本的已知问题，请参见 [GitHub](https://github.com/rancher/rancher/releases) 中的发行说明，或查看 [Rancher 论坛](https://forums.rancher.com/c/announcements/12)。 不支持 _升级_ 或 _升级到_ [rancher-alpha 仓库](../resources/choose-a-rancher-version.md#helm-chart-仓库)中的任何 Chart。 + ### Helm 版本 + + 本安装指南假定你使用的是 Helm 3。 如果你使用 Helm 2，请参见 [Helm 2 迁移到 Helm 3 文档](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/)。如果你不能升级到 Helm 3，[Helm 2 升级页面](/versioned_docs/version-2.0-2.4/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades/helm2.md)提供了使用 Helm 2 升级的旧升级指南。 ### 离线安装：推送镜像到私有镜像仓库 -[仅适用于离线安装](../../../pages-for-subheaders/air-gapped-helm-cli-install.md)：为新的 Rancher Server 版本收集和推送镜像。使用你需要针对 Rancher 版本升级的镜像，按照步骤[推送镜像到私有镜像仓库](../other-installation-methods/air-gapped-helm-cli-install/publish-images.md)。 +[仅适用于离线安装](../other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)：为新的 Rancher Server 版本收集和推送镜像。使用你需要针对 Rancher 版本升级的镜像，按照步骤[推送镜像到私有镜像仓库](../other-installation-methods/air-gapped-helm-cli-install/publish-images.md)。 ### 使用 cert-manager 0.8.0 之前的版本升级 @@ -42,7 +48,6 @@ kubeconfig 也可以通过 `--kubeconfig` 标签（详情请参见 https://helm. 按照以下步骤升级 Rancher Server： - ### 1. 备份运行 Rancher Server 的 Kubernetes 集群 使用[备份应用](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md)来备份 Rancher。 @@ -59,21 +64,21 @@ kubeconfig 也可以通过 `--kubeconfig` 标签（详情请参见 https://helm. 1. 获取你用来安装 Rancher 的仓库名称。 - 关于仓库及其区别，请参见 [Helm Chart Repositories](../resources/choose-a-rancher-version.md#helm-chart-仓库)。 + 关于仓库及其区别，请参见 [Helm Chart 仓库](../resources/choose-a-rancher-version.md#helm-chart-仓库)。 - Latest：建议用于试用最新功能 - ``` - helm repo add rancher-latest https://releases.rancher.com/server-charts/latest - ``` + ``` + helm repo add rancher-latest https://releases.rancher.com/server-charts/latest + ``` - Stable：建议用于生产环境 - ``` - helm repo add rancher-stable https://releases.rancher.com/server-charts/stable - ``` + ``` + helm repo add rancher-stable https://releases.rancher.com/server-charts/stable + ``` - Alpha：即将发布的实验性预览。 - ``` - helm repo add rancher-alpha https://releases.rancher.com/server-charts/alpha - ``` - 注意：不支持升级到 Alpha 版、从 Alpha 版升级或在 Alpha 版之间升级。 + ``` + helm repo add rancher-alpha https://releases.rancher.com/server-charts/alpha + ``` + 注意：不支持升级到 Alpha 版、从 Alpha 版升级或在 Alpha 版之间升级。 ``` helm repo list @@ -96,6 +101,7 @@ kubeconfig 也可以通过 `--kubeconfig` 标签（详情请参见 https://helm. ```plain helm fetch rancher-/rancher ``` + 你可以通过 `--version=` 标记，来指定要升级的目标 Chart 版本。例如： ```plain @@ -112,7 +118,6 @@ kubeconfig 也可以通过 `--kubeconfig` 标签（详情请参见 https://helm. ::: - 从当前安装的 Rancher Helm Chart 中获取用 `--set`传递的值。 ``` @@ -128,13 +133,16 @@ hostname: rancher.my.org ::: :::tip + Deployment 的名称可能会有所不同。例如，如果你通过 AWS Marketplace 部署 Rancher，则 Deployment 的名称为“rancher-stable”。 因此： + ``` helm get values rancher-stable -n cattle-system hostname: rancher.my.org ``` + ::: 如果要将 cert-manager 从 v1.5 或更早的版本升级到最新版本，请参阅 [cert-manager upgrade docs](../resources/upgrade-cert-manager.md#选项-c升级-15-及以下版本的-cert-manager) 了解如何在不卸载或重新安装 Rancher 的情况下升级 cert-manager。否则，请按照以下[ Rancher 升级步骤](#rancher-升级步骤)进行操作。 @@ -160,13 +168,16 @@ helm upgrade rancher rancher-/rancher \ ::: :::tip + 如果你通过 AWS Marketplace 部署 Rancher，则 Deployment 的名称为“rancher-stable”。 因此： + ``` helm upgrade rancher-stable rancher-/rancher \ --namespace cattle-system \ --set hostname=rancher.my.org ``` + ::: 另外，你也可以将当前的值导出到一个文件中，并在升级时引用该文件。例如，如果你只需要改变 Rancher 的版本： diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-and-upgrade.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-and-upgrade.md new file mode 100644 index 00000000000..a0568b31e53 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-and-upgrade.md @@ -0,0 +1,98 @@ +--- +title: 安装/升级 Rancher +description: 了解如何在开发和生产环境中安装 Rancher。了解单节点和高可用安装 +--- + + + + + +本节介绍了 Rancher 各种安装方式以及每个安装方式的优点。 + +## 名词解释 + +本章节涉及以下名词： + +- **Rancher Server**：用于管理和配置 Kubernetes 集群。你可以通过 Rancher Server 的 UI 与下游 Kubernetes 集群进行交互。Rancher Management Server 可以安装到任意 Kubernetes 集群上，包括托管的集群，如 Amazon EKS 集群。 +- **RKE（Rancher Kubernetes Engine）**：是经过认证的 Kubernetes 发行版，也是用于创建和管理 Kubernetes 集群的 CLI 工具和库。 +- **K3s（轻量级 Kubernetes）**：也是经过认证的 Kubernetes 发行版。它比 RKE 更新，更易用且更轻量，其所有组件都在一个小于 100 MB 的二进制文件中。 +- **RKE2**：一个完全合规的 Kubernetes 发行版，专注于安全和合规性。 + +`restrictedAdmin` Helm Chart 选项在 **Rancher Server** 可用。如果该选项设置为 true，初始的 Rancher 用户访问本地 Kubernetes 集群会受到限制，以避免权限升级。详情请参见 [restricted-admin 角色](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md#受限管理员)。 + +## 安装方式概述 + +Rancher 可以安装在以下主要架构上： + +### 使用 Helm CLI 安装的高可用 Kubernetes + +我们建议使用 Kubernetes 包管理器 Helm 在专用的 Kubernetes 集群上安装 Rancher。在 RKE 集群中，需要使用三个节点才能实现高可用集群。在 K3s 集群中，只需要两个节点即可。 + +### 通过 AWS Marketplace 在 EKS 上安装 Rancher + +你可以[通过 AWS Marketplace](../quick-start-guides/deploy-rancher-manager/aws-marketplace.md) 将 Rancher 安装到 Amazon Elastic Kubernetes Service (EKS) 上。部署的 EKS 集群已生产就绪，并遵循 AWS 最佳实践。 + +### 单节点 Kubernetes 安装 + +Rancher 可以安装在单节点 Kubernetes 集群上。但是，在单节点安装的情况下，Rancher Server 没有高可用性。而高可用性对在生产环境中运行 Rancher 非常重要。 + +但是，如果你想要短期内使用单节点节省资源，同时又保留高可用性迁移路径，那么单节点 Kubernetes 安装也是合适的。你也可以之后向集群中添加节点，获得高可用的 Rancher Server。 + +### Docker 安装 + +如果你的目的是测试或演示，你可以使用 Docker 把 Rancher 安装到单个节点中。本地 Kubernetes 集群是安装到单个 Docker 容器中的，而 Rancher 是安装到本地集群中的。 + +Rancher backup operator 可将 Rancher 从单个 Docker 容器迁移到高可用 Kubernetes 集群上。详情请参见[把 Rancher 迁移到新集群](../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)。 + +### 其他方式 + +如果你需要在离线环境中或使用 HTTP 代理安装 Rancher，请参见以下独立的说明文档： + +| 网络访问方式 | 基于 Kubernetes 安装（推荐） | 基于 Docker 安装 | +| ---------------- | ----------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 可直接访问互联网 | [文档](install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md) | [文档](other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md) | +| 使用 HTTP 代理 | [文档](other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md) | [文档](other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md)及[配置](../../reference-guides/single-node-rancher-in-docker/http-proxy-configuration.md) | +| 离线环境 | [文档](other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md) | [文档](other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md) | + +我们建议在 Kubernetes 集群上安装 Rancher，因为在多节点集群中，Rancher Server 可以实现高可用。高可用配置可以提升 Rancher 访问其管理的下游 Kubernetes 集群的稳定性。 + +因此，我们建议在生产级别的架构中，设置一个高可用的 Kubernetes 集群，然后在这个集群上安装 Rancher。安装 Rancher 后，你可以使用 Rancher 部署和管理 Kubernetes 集群。 + +如果你的目的是测试或演示，你可以将 Rancher 安装到单个 Docker 容器中。Docker 安装可以让你实现开箱即用，以使用 Rancher 设置 Kubernetes 集群。Docker 安装主要是用于探索 Rancher Server 的功能，只适用于开发和测试。 + +[在 Kubernetes 上安装 Rancher 的说明](install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)介绍了如何首先使用 K3s 或 RKE 创建和管理 Kubernetes 集群，然后再将 Rancher 安装到该集群上。 + +如果 Kubernetes 集群中的节点正在运行且满足[节点要求](installation-requirements/installation-requirements.md)，你可以使用 Helm 将 Rancher 部署到 Kubernetes 上。Helm 使用 Rancher 的 Helm Chart 在 Kubernetes 集群的每个节点上安装 Rancher 的副本。我们建议使用负载均衡器将流量定向到集群中的每个 Rancher 副本上。 + +如需进一步了解 Rancher 架构，请参见[架构概述](../../reference-guides/rancher-manager-architecture/rancher-manager-architecture.md)，[生产级别架构推荐](../../reference-guides/rancher-manager-architecture/architecture-recommendations.md)或[最佳实践指南](../../reference-guides/best-practices/rancher-server/tips-for-running-rancher.md)。 + +## 先决条件 + +安装 Rancher 之前，请确保你的节点满足所有[安装要求](installation-requirements/installation-requirements.md)。 + +## 架构建议 + +为了达到最佳性能和安全性，我们建议你为 Rancher Management Server 使用单独的专用 Kubernetes 集群。不建议在此集群上运行用户工作负载。部署 Rancher 后，你可以[创建或导入集群](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)来运行你的工作负载。 + +详情请参见[架构推荐](../../reference-guides/rancher-manager-architecture/architecture-recommendations.md)。 + +### 在 Kubernetes 上安装 Rancher 的更多选项 + +参见 [Helm Chart 选项](installation-references/helm-chart-options.md)以了解在 Kubernetes 集群上安装 Rancher 的其他配置，包括： + +- [开启 API 审计日志来记录所有事务](installation-references/helm-chart-options.md#api-审计日志) +- [负载均衡器上的 TLS 终止](installation-references/helm-chart-options.md#外部-tls-终止) +- [自定义 Ingress](installation-references/helm-chart-options.md#自定义-ingress) + +在 Rancher 的安装指南中，我们推荐使用 K3s 或 RKE 来配置 Kubernetes 集群，然后再在这个集群中安装 Rancher。K3s 和 RKE 均提供许多配置选项，用于为你的具体环境自定义 Kubernetes 集群。有关选项和功能的完整列表，请参见： + +- [RKE 配置选项](https://rancher.com/docs/rke/latest/en/config-options/) +- [K3s 配置选项](https://rancher.com/docs/k3s/latest/en/installation/install-options/) + +### 在 Docker 上安装 Rancher 的更多选项 + +参见 [Docker 安装选项](other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md)了解其他配置，包括： + +- [开启 API 审计日志来记录所有事务](../../reference-guides/single-node-rancher-in-docker/advanced-options.md#api-审计日志) +- [外部负载均衡器](../../how-to-guides/advanced-user-guides/configure-layer-7-nginx-load-balancer.md) +- [持久化数据存储](../../reference-guides/single-node-rancher-in-docker/advanced-options.md#持久化数据) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-references/feature-flags.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-references/feature-flags.md index 873a0689b14..b6d10b82311 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-references/feature-flags.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-references/feature-flags.md @@ -2,9 +2,13 @@ title: 功能开关 --- + + + + 使用功能开关（Feature Flag），你可以试用可选或实验性的功能并启用正在逐步淘汰的旧版功能。 -要了解功能的值以及如何启用它们，请参阅[启用实验性功能](../../../pages-for-subheaders/enable-experimental-features.md)。 +要了解功能的值以及如何启用它们，请参阅[启用实验性功能](../../../how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 :::note @@ -16,7 +20,7 @@ title: 功能开关 - `continuous-delivery`：允许从 Fleet 中单独禁用 Fleet GitOps。有关详细信息，请参阅[持续交付](../../../how-to-guides/advanced-user-guides/enable-experimental-features/continuous-delivery.md)。 - `fleet`：v2.6 及更高版本的 Rancher 配置框架需要 Fleet。即使你在旧 Rancher 版本中禁用了该标志，该标志也将在升级时自动启用。有关详细信息，请参阅 [Fleet - GitOps at Scale](../../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md)。 -- `harvester`：管理 Virtualization Management 页面的访问。用户可以在该页面直接导航到 Harvester 集群并访问 Harvester UI。有关详细信息，请参阅 [Harvester 集成](../../../integrations-in-rancher/harvester.md)。 +- `harvester`：管理 Virtualization Management 页面的访问。用户可以在该页面直接导航到 Harvester 集群并访问 Harvester UI。有关详细信息，请参阅 [Harvester 集成](../../../integrations-in-rancher/harvester/overview.md)。 - `istio-virtual-service-ui`：启用[可视界面](../../../how-to-guides/advanced-user-guides/enable-experimental-features/istio-traffic-management-features.md)来创建、读取、更新和删除 Istio 虚拟服务和目标规则，这些都是 Istio 流量管理功能。 - `legacy`：启用 2.5.x 及更早版本的一组功能，这些功能正逐渐被新的实现淘汰。它们是已弃用以及后续可用于新版本的功能组合。新的 Rancher 安装会默认禁用此标志。如果你从以前版本的 Rancher 升级，此标志会启用。 - `multi-cluster-management`：允许配置和管理多个 Kubernetes 集群。此标志只能在安装时设置。后续无法启用或禁用它。 @@ -27,13 +31,13 @@ title: 功能开关 下表介绍了 Rancher 中功能开关的可用性和默认值。标记为“GA”的功能已普遍可用： -| 功能开关名称 | 默认值 | 状态 | 可用于 | -| ----------------------------- | ------------- | ------------ | --------------- | -| `continuous-delivery` | `true` | GA | v2.6.0 | -| `fleet` | `true` | 不能禁用 | v2.6.0 | -| `fleet` | `true` | GA | v2.5.0 | -| `harvester` | `true` | 实验功能 | v2.6.1 | -| `legacy` | 新安装：`false`；升级：`true` | GA | v2.6.0 | -| `rke1-custom-node-cleanup` | `true` | GA | v2.6.0 | -| `rke2` | `true` | 实验功能 | v2.6.0 | -| `token-hashing` | 新安装：`false`；升级：`true` | GA | v2.6.0 | +| 功能开关名称 | 默认值 | 状态 | 可用于 | +| -------------------------- | ----------------------------- | -------- | ------ | +| `continuous-delivery` | `true` | GA | v2.6.0 | +| `fleet` | `true` | 不能禁用 | v2.6.0 | +| `fleet` | `true` | GA | v2.5.0 | +| `harvester` | `true` | 实验功能 | v2.6.1 | +| `legacy` | 新安装：`false`；升级：`true` | GA | v2.6.0 | +| `rke1-custom-node-cleanup` | `true` | GA | v2.6.0 | +| `rke2` | `true` | 实验功能 | v2.6.0 | +| `token-hashing` | 新安装：`false`；升级：`true` | GA | v2.6.0 | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index bd69a89cfee..026750565a8 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -1,65 +1,76 @@ --- title: Rancher Helm Chart 选项 -keywords: [rancher helm chart, rancher helm 选项, rancher helm chart 选项, helm chart rancher, helm 选项 rancher, helm chart 选项 rancher] +keywords: + [ + rancher helm chart, + rancher helm 选项, + rancher helm chart 选项, + helm chart rancher, + helm 选项 rancher, + helm chart 选项 rancher, + ] --- + + + + 本文提供了 Rancher Helm Chart 的配置参考。 如需选择 Helm Chart 版本，请参见[本页](../../../getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md)。 -了解开启实验性功能的详情，请参见[本页](../../../pages-for-subheaders/enable-experimental-features.md)。 +了解开启实验性功能的详情，请参见[本页](../../../how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 ## 常用选项 -| 选项 | 默认值 | 描述 | -| ------------------------- | ------------- | ---------------------------------------------------------------------------------- | -| `bootstrapPassword` | " " | `string` - 为第一个管理员用户设置[引导密码](#引导密码)。登录后，管理员需要重置密码。如不设置，会使用随机生成的引导密码。 | -| `hostname` | " " | `string` - 你的 Rancher Server 的完全限定的域名（FQDN) | -| `ingress.tls.source` | "rancher" | `string` - 从哪里获取 Ingress 的证书- "rancher, letsEncrypt, secret" | -| `letsEncrypt.email` | " " | `string` - 你的邮箱地址 | -| `letsEncrypt.environment` | "production" | `string` - 可选项："staging, production" | -| `privateCA` | false | `bool` - 如果你的证书是由私有 CA 签发的，把这个值设置为 true | +| 选项 | 默认值 | 描述 | +| ------------------------- | ------------ | ------------------------------------------------------------------------------------------------------------------------ | +| `bootstrapPassword` | " " | `string` - 为第一个管理员用户设置[引导密码](#引导密码)。登录后，管理员需要重置密码。如不设置，会使用随机生成的引导密码。 | +| `hostname` | " " | `string` - 你的 Rancher Server 的完全限定的域名（FQDN) | +| `ingress.tls.source` | "rancher" | `string` - 从哪里获取 Ingress 的证书- "rancher, letsEncrypt, secret" | +| `letsEncrypt.email` | " " | `string` - 你的邮箱地址 | +| `letsEncrypt.environment` | "production" | `string` - 可选项："staging, production" | +| `privateCA` | false | `bool` - 如果你的证书是由私有 CA 签发的，把这个值设置为 true |
## 高级选项 -| 选项 | 默认值 | 描述 | -| ------------------------------ | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -| `additionalTrustedCAs` | false | `bool` - 请参见[额外的授信 CA](#额外的授信-ca) | -| `addLocal` | "true" | `string` - 让 Rancher 检测并导入 “local” Rancher Server 集群。_注意：此选项在 2.5.0 中已不可用。你可考虑使用 `restrictedAdmin` 选项，来避免用户修改本地集群。_ | -| `antiAffinity` | "preferred" | `string` - Rancher Pod 的反亲和性规则 - "preferred, required" | -| `auditLog.destination` | "sidecar" | `string` - 发送审计日志到 Sidecar 容器的控制台或 hostPath 卷 - "sidecar, hostPath" | -| `auditLog.hostPath` | "/var/log/rancher/audit" | `string` - 主机上的日志文件目标地址（仅当`auditLog.destination` 的值是 `hostPath` 时生效） | -| `auditLog.level` | 0 | `int` - 设置 [API 审计日志](../../../how-to-guides/advanced-user-guides/enable-api-audit-log.md)等级。0 代表关闭。[0-3] | -| `auditLog.maxAge` | 1 | `int` - 旧审计日志文件最多可保留的天数（仅当`auditLog.destination` 的值是 `hostPath` 时生效） | -| `auditLog.maxBackup` | 1 | `int` - 审计文件最大可保留的个数（仅当 `auditLog.destination` 的值是 `hostPath` 时生效） | -| `auditLog.maxSize` | 100 | `int` - 在审计日志被轮换前的最大容量，单位是 MB（仅当 `auditLog.destination` 的值是 `hostPath` 时生效） | -| `auditLog.image.repository` | "registry.suse.com/bci/bci-micro" | `string` - 用于收集审计日志的镜像的位置。 | -| `auditLog.image.tag` | "15.4.14.3" | `string` - 用于收集审计日志的镜像的标签。 | -| `auditLog.image.pullPolicy` | "IfNotPresent" | `string` - 覆盖 auditLog 镜像的 imagePullPolicy - “Always”、“Never”、“IfNotPresent”。 | -| `busyboxImage` | "" | `string` - 用于收集审计日志的 busybox 镜像位置。_注意：此选项已弃用，请使用 `auditLog.image.repository` 来控制审计 sidecar 镜像_。 | -| `certmanager.version` | "" | `string` - 设置 cert-manager compatibility | -| `debug` | false | `bool` - 在 Rancher Server 设置 debug 参数 | -| `extraEnv` | [] | `list` - 为 Rancher 额外设置环境变量 | -| `imagePullSecrets` | [] | `list` - 私有镜像仓库凭证的密文名称列表 | -| `ingress.configurationSnippet` | "" | `string` - 添加额外的 Nginx 配置。可用于代理配置。 | -| `ingress.extraAnnotations` | {} | `map` - 用于自定义 Ingress 的额外注释 | -| `ingress.enabled` | true | 如果值为 false，Helm 不会安装 Rancher Ingress。你可把值设为 false 以部署你自己的 Ingress。 | -| `letsEncrypt.ingress.class` | "" | `string` - cert-manager acmesolver ingress 的可选 ingress 类，用于响应 Let's Encrypt ACME 质询。选项：traefik，nginx。 | | -| `noProxy` | "127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local,cattle-system.svc" | `string` - 不使用代理的主机名或 IP 地址的逗号分隔列表 | | -| `proxy` | "" | `string` - 给 Rancher 配置的 HTTP[S] 代理 | -| `rancherImage` | "rancher/rancher" | `string` - Rancher 镜像源 | -| `rancherImagePullPolicy` | "IfNotPresent" | `string` - 覆盖 Rancher Server 镜像的 imagePullPolicy - "Always", "Never", "IfNotPresent" | -| `rancherImageTag` | 和 Chart 版本一致 | `string` - rancher/rancher 镜像标签 | -| `replicas` | 3 | `int` - Rancher Server 副本数。如果设为 -1，会根据集群中的可用节点数自动选择 1，2或3。 | -| `resources` | {} | `map` - Rancher Pod 资源请求和限制 | -| `restrictedAdmin` | `false` | `bool` - 如果值为 true，初始的 Rancher 用户访问本地 Kubernetes 集群会受到限制，以避免权限升级。详情请参见 [restricted-admin 角色](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md#受限管理员)。 | -| `systemDefaultRegistry` | "" | `string` - 用于所有系统容器镜像的私有仓库，例如 http://registry.example.com/ | -| `tls` | "ingress" | `string` - 详情请参见[外部 TLS 终止](#外部-tls-终止)。- "ingress, external" | -| `useBundledSystemChart` | `false` | `bool` - 选择 Rancher Server 打包的 system-charts。此参数用于离线环境安装。 | -| `global.cattle.psp.enabled` | `true` | `bool` - 使用 Rancher v2.7.2-v2.7.4 时，选择 `false` 以禁用 Kubernetes v1.25 及更高版本的 PSP。使用 Rancher v2.7.5 及更高版本时，Rancher 会尝试检测集群是否运行不支持 PSP 的 Kubernetes 版本，如果确定集群不支持 PSP，则将默认 PSP 的使用设置为 false。你仍然可以通过显式提供此值的 `true` 或 `false` 来手动覆盖此值。在支持 PSP 的集群中（例如使用 Kubernetes v1.24 或更低版本的集群），Rancher 仍将默认使用 PSP。 | - +| 选项 | 默认值 | 描述 | +| ------------------------------ | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | +| `additionalTrustedCAs` | false | `bool` - 请参见[额外的授信 CA](#额外的授信-ca) | +| `addLocal` | "true" | `string` - 让 Rancher 检测并导入 “local” Rancher Server 集群。_注意：此选项在 2.5.0 中已不可用。你可考虑使用 `restrictedAdmin` 选项，来避免用户修改本地集群。_ | +| `antiAffinity` | "preferred" | `string` - Rancher Pod 的反亲和性规则 - "preferred, required" | +| `auditLog.destination` | "sidecar" | `string` - 发送审计日志到 Sidecar 容器的控制台或 hostPath 卷 - "sidecar, hostPath" | +| `auditLog.hostPath` | "/var/log/rancher/audit" | `string` - 主机上的日志文件目标地址（仅当`auditLog.destination` 的值是 `hostPath` 时生效） | +| `auditLog.level` | 0 | `int` - 设置 [API 审计日志](../../../how-to-guides/advanced-user-guides/enable-api-audit-log.md)等级。0 代表关闭。[0-3] | +| `auditLog.maxAge` | 1 | `int` - 旧审计日志文件最多可保留的天数（仅当`auditLog.destination` 的值是 `hostPath` 时生效） | +| `auditLog.maxBackup` | 1 | `int` - 审计文件最大可保留的个数（仅当 `auditLog.destination` 的值是 `hostPath` 时生效） | +| `auditLog.maxSize` | 100 | `int` - 在审计日志被轮换前的最大容量，单位是 MB（仅当 `auditLog.destination` 的值是 `hostPath` 时生效） | +| `auditLog.image.repository` | "registry.suse.com/bci/bci-micro" | `string` - 用于收集审计日志的镜像的位置。 | +| `auditLog.image.tag` | "15.4.14.3" | `string` - 用于收集审计日志的镜像的标签。 | +| `auditLog.image.pullPolicy` | "IfNotPresent" | `string` - 覆盖 auditLog 镜像的 imagePullPolicy - “Always”、“Never”、“IfNotPresent”。 | +| `busyboxImage` | "" | `string` - 用于收集审计日志的 busybox 镜像位置。_注意：此选项已弃用，请使用 `auditLog.image.repository` 来控制审计 sidecar 镜像_。 | +| `certmanager.version` | "" | `string` - 设置 cert-manager compatibility | +| `debug` | false | `bool` - 在 Rancher Server 设置 debug 参数 | +| `extraEnv` | [] | `list` - 为 Rancher 额外设置环境变量 | +| `imagePullSecrets` | [] | `list` - 私有镜像仓库凭证的密文名称列表 | +| `ingress.configurationSnippet` | "" | `string` - 添加额外的 Nginx 配置。可用于代理配置。 | +| `ingress.extraAnnotations` | {} | `map` - 用于自定义 Ingress 的额外注释 | +| `ingress.enabled` | true | 如果值为 false，Helm 不会安装 Rancher Ingress。你可把值设为 false 以部署你自己的 Ingress。 | +| `letsEncrypt.ingress.class` | "" | `string` - cert-manager acmesolver ingress 的可选 ingress 类，用于响应 Let's Encrypt ACME 质询。选项：traefik，nginx。 | | +| `noProxy` | "127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local,cattle-system.svc" | `string` - 不使用代理的主机名或 IP 地址的逗号分隔列表 | | +| `proxy` | "" | `string` - 给 Rancher 配置的 HTTP[S] 代理 | +| `rancherImage` | "rancher/rancher" | `string` - Rancher 镜像源 | +| `rancherImagePullPolicy` | "IfNotPresent" | `string` - 覆盖 Rancher Server 镜像的 imagePullPolicy - "Always", "Never", "IfNotPresent" | +| `rancherImageTag` | 和 Chart 版本一致 | `string` - rancher/rancher 镜像标签 | +| `replicas` | 3 | `int` - Rancher Server 副本数。如果设为 -1，会根据集群中的可用节点数自动选择 1，2 或 3。 | +| `resources` | {} | `map` - Rancher Pod 资源请求和限制 | +| `restrictedAdmin` | `false` | `bool` - 如果值为 true，初始的 Rancher 用户访问本地 Kubernetes 集群会受到限制，以避免权限升级。详情请参见 [restricted-admin 角色](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md#受限管理员)。 | +| `systemDefaultRegistry` | "" | `string` - 用于所有系统容器镜像的私有仓库，例如 http://registry.example.com/ | +| `tls` | "ingress" | `string` - 详情请参见[外部 TLS 终止](#外部-tls-终止)。- "ingress, external" | +| `useBundledSystemChart` | `false` | `bool` - 选择 Rancher Server 打包的 system-charts。此参数用于离线环境安装。 | +| `global.cattle.psp.enabled` | `true` | `bool` - 使用 Rancher v2.7.2-v2.7.4 时，选择 `false` 以禁用 Kubernetes v1.25 及更高版本的 PSP。使用 Rancher v2.7.5 及更高版本时，Rancher 会尝试检测集群是否运行不支持 PSP 的 Kubernetes 版本，如果确定集群不支持 PSP，则将默认 PSP 的使用设置为 false。你仍然可以通过显式提供此值的 `true` 或 `false` 来手动覆盖此值。在支持 PSP 的集群中（例如使用 Kubernetes v1.24 或更低版本的集群），Rancher 仍将默认使用 PSP。 | ### 引导密码 @@ -81,13 +92,13 @@ kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ 启用 [API 审计日志](../../../how-to-guides/advanced-user-guides/enable-api-audit-log.md)。 -你可以像收集其他容器日志一样收集此日志。在 Rancher Server 集群上为 `System` 项目启用 [Logging](../../../pages-for-subheaders/logging.md)。 +你可以像收集其他容器日志一样收集此日志。在 Rancher Server 集群上为 `System` 项目启用 [Logging](../../../integrations-in-rancher/logging/logging.md)。 ```plain --set auditLog.level=1 ``` -默认情况下，启用审计日志会在 Rancher pod 中创建一个 Sidecar 容器。这个容器（`rancher-audit-log`）会把日志流传输到 `stdout`。你可以像收集其他容器日志一样收集此日志。如果你使用 Sidecar 作为审计日志的目标时， `hostPath`，`maxAge`，`maxBackups` 和 `maxSize` 选项不会生效。建议使用你的操作系统或 Docker Daemon 的日志轮换功能来控制磁盘空间的使用。请为 Rancher Server 集群或 System 项目启用 [Logging](../../../pages-for-subheaders/logging.md)。 +默认情况下，启用审计日志会在 Rancher pod 中创建一个 Sidecar 容器。这个容器（`rancher-audit-log`）会把日志流传输到 `stdout`。你可以像收集其他容器日志一样收集此日志。如果你使用 Sidecar 作为审计日志的目标时， `hostPath`，`maxAge`，`maxBackups` 和 `maxSize` 选项不会生效。建议使用你的操作系统或 Docker Daemon 的日志轮换功能来控制磁盘空间的使用。请为 Rancher Server 集群或 System 项目启用 [Logging](../../../integrations-in-rancher/logging/logging.md)。 将 `auditLog.destination` 的值设为 `hostPath`，可以将日志转发到与主机系统共享的卷，而不是传输到 Sidecar 容器。如果目标设置为 `hostPath`，你可能需要调整其他 auditLog 参数以进行日志轮换。 @@ -151,21 +162,23 @@ kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{ Rancher 的一些功能（Helm Chart）需要使用互联网才能使用。你可以使用 `proxy` 设置代理服务器，或使用 `extraEnv` 设置 `HTTPS_PROXY` 环境变量来指向代理服务器。 将要排除的 IP 使用逗号分隔列表添加到 `noProxy` Chart value 中。确保添加了以下值： + - Pod 集群 IP 范围（默认值：`10.42.0.0/16`）。 - Service Cluster IP 范围（默认值：`10.43.0.0/16`）。 - 内部集群域（默认值：`.svc,.cluster.local`）。 - 任何 Worker 集群 `controlplane` 节点。 - Rancher 支持在此列表中使用 CIDR 表示法来表示范围。 + Rancher 支持在此列表中使用 CIDR 表示法来表示范围。 不包括敏感数据时，可以使用 `proxy` 或 `extraEnv` Chart 选项。使用 `extraEnv` 时将忽略 `noProxy` Helm 选项。因此，`NO_PROXY` 环境变量也必须设置为 `extraEnv`。 -以下是使用 `extraEnv` Chart 选项设置代理的示例： +以下是使用 `proxy` Chart 选项设置代理的示例： ```plain --set proxy="http:///" ``` 使用 `extraEnv` Chart 选项设置代理的示例： + ```plain --set extraEnv[1].name=HTTPS_PROXY --set extraEnv[1].value="http://:/" @@ -176,6 +189,7 @@ Rancher 的一些功能（Helm Chart）需要使用互联网才能使用。你 包含敏感数据（例如代理认证凭证）时，请使用 `extraEnv` 选项和 `valueFrom.secretRef` 来防止敏感数据在 Helm 或 Rancher 部署中暴露。 下面是使用 `extraEnv` 配置代理的示例。此示例 Secret 在 Secret 的 `"https-proxy-url"` 键中包含 `"http://:@:/"` 值： + ```plain --set extraEnv[1].name=HTTPS_PROXY --set extraEnv[1].valueFrom.secretKeyRef.name=secret-name @@ -202,7 +216,7 @@ kubectl -n cattle-system create secret generic tls-ca-additional --from-file=ca- ### 私有仓库和离线安装 -有关使用私有仓库安装 Rancher 的详情，请参见[离线安装](../../../pages-for-subheaders/air-gapped-helm-cli-install.md)。 +有关使用私有仓库安装 Rancher 的详情，请参见[离线安装](../other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)。 ## 外部 TLS 终止 @@ -226,7 +240,7 @@ kubectl -n cattle-system create secret generic tls-ca-additional --from-file=ca- ingress: provider: nginx options: - use-forwarded-headers: 'true' + use-forwarded-headers: "true" ``` ### 必须的 Header diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-references/installation-references.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-references/installation-references.md new file mode 100644 index 00000000000..5174e27028c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-references/installation-references.md @@ -0,0 +1,9 @@ +--- +title: 安装参考 +--- + + + + + +有关其他安装资源，请参阅以下参考指南：[Rancher Helm Chart 选项](helm-chart-options.md)、[TLS 设置](tls-settings.md)和[功能开关](feature-flags.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-requirements/install-docker.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-requirements/install-docker.md index 8b98ce09030..e03b752d4cc 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-requirements/install-docker.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-requirements/install-docker.md @@ -2,22 +2,26 @@ title: 安装 Docker --- -在使用 Helm 或 Docker 在 RKE 集群节点上安装 Rancher Server 前，你需要先安装 Docker。RKE2 和 K3s 集群不要求使用 Docker。 + + + + +在使用 Helm 在 RKE 集群节点上或使用 Docker 安装 Rancher Server 前，你需要在节点中先安装 Docker。RKE2 和 K3s 集群不要求使用 Docker。 Docker 有几个安装方法。一种方法是参见 [Docker 官方文档](https://docs.docker.com/install/)以了解如何在 Linux 上安装 Docker。不同 Linux 发行版的安装步骤可能有所不同。 -另一种方式是使用 Rancher 的 Docker 安装脚本，该脚本可用于较新的 Docker 版本。 +另一种方式是使用 Rancher 的 Docker 安装脚本，该脚本可用于较新的 Docker 版本。 Rancher 为每个 Kubernetes 支持的上游 Docker 版本提供了安装脚本。 -例如，此命令可用于在 SUSE Linux Enterprise 或 Ubuntu 等主要 Linux 发行版上安装 Docker 20.10： +例如，此命令可用于在 SUSE Linux Enterprise 或 Ubuntu 等主要 Linux 发行版上安装 Docker ： -``` -curl https://releases.rancher.com/install-docker/20.10.sh | sh +```bash +curl https://releases.rancher.com/install-docker/.sh | sh ``` -Rancher 提供 Kubernetes 支持的所有上游 Docker 版本的安装脚本。如需了解我们是否提供某个 Docker 版本的安装脚本，请参见包含了 Rancher 所有的 Docker 安装脚本的 [GitHub 仓库](https://github.com/rancher/install-docker)。 +请参阅 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix)，使用匹配你的操作系统和 Rancher 版本并且经过验证的 Docker 版本。 尽管支持矩阵列出了经过验证的 Docker 版本直至补丁版本，但只有发行版的主要版本和次要版本与 Docker 安装脚本相关。 请注意，必须应用以下 sysctl 设置： -``` +```bash net.bridge.bridge-nf-call-iptables=1 ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md new file mode 100644 index 00000000000..51574288460 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md @@ -0,0 +1,221 @@ +--- +title: 安装要求 +description: 如果 Rancher 配置在 Docker 或 Kubernetes 中运行时，了解运行 Rancher Server 的每个节点的节点要求 +--- + + + + + +本文描述了对需要安装 Rancher Server 的节点的软件、硬件和网络要求。Rancher Server 可以安装在单个节点或高可用的 Kubernetes 集群上。 + +:::note 重要提示： + +如果你需要在 Kubernetes 集群上安装 Rancher，该节点的要求与用于运行应用和服务的[下游集群的节点要求](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)不同。 + +::: + +Rancher UI 在基于 Firefox 或 Chromium 的浏览器（Chrome、Edge、Opera、Brave）中效果最佳。 + +查看我们的[最佳实践](../../../reference-guides/best-practices/rancher-server/tips-for-running-rancher.md)页面，获取在生产环境中运行 Rancher Server 的建议。 + +## Kubernetes 与 Rancher 的兼容性 + +Rancher 需要安装在支持的 Kubernetes 版本上。请查阅 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions)，确保你的 Kubernetes 版本受支持。 + +### 在加固的 Kubernetes 集群上安装 Rancher + +如果在加固的 Kubernetes 集群上安装 Rancher，请参阅[豁免必须的 Rancher 命名空间](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md#豁免必须的-rancher-命名空间)部分了解更详细的要求。 + +## 操作系统和容器运行时要求 + +所有支持的操作系统都使用 64-bit x86 架构。Rancher 兼容当前所有的主流 Linux 发行版。 + +[Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions)列出了每个 Rancher 版本测试过的操作系统和 Docker 版本。 + +运行 RKE 集群的节点需要安装 Docker。RKE2 或 K3s 集群不需要它。 + +请安装 `ntp`（Network Time Protocol），以防止在客户端和服务器之间由于时间不同步造成的证书验证错误。 + +某些 Linux 发行版的默认防火墙规则可能会阻止 Kubernetes 集群内的通信。从 Kubernetes v1.19 开始，你必须关闭 firewalld，因为它与 Kubernetes 网络插件冲突。 + +如果你不太想这样做的话，你可以查看[相关问题](https://github.com/rancher/rancher/issues/28840)中的建议。某些用户已能成功[使用 ACCEPT 策略 为 Pod CIDR 创建一个独立的 firewalld 区域](https://github.com/rancher/rancher/issues/28840#issuecomment-787404822)。 + +如果你需要在 ARM64 上使用 Rancher，请参见[在 ARM64（实验功能）上运行 Rancher](../../../how-to-guides/advanced-user-guides/enable-experimental-features/rancher-on-arm64.md)。 + +### RKE2 要求 + +对于容器运行时，RKE2 附带了自己的 containerd。RKE2 安装不需要 Docker。 + +如需了解 RKE2 通过了哪些操作系统版本的测试，请参见 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions)。 + +### K3s 要求 + +对于容器运行时，K3s 默认附带了自己的 containerd。你也可以将 K3s 配置为使用已安装的 Docker 运行时。有关在 Docker 中使用 K3s 的更多信息，请参阅 [K3s 文档](https://docs.k3s.io/advanced#using-docker-as-the-container-runtime)。 + +Rancher 需要安装在支持的 Kubernetes 版本上。如需了解你使用的 Rancher 版本支持哪些 Kubernetes 版本，请参见 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions)。如需指定 K3s 版本，在运行 K3s 安装脚本时，使用 `INSTALL_K3S_VERSION` 环境变量。 + +如果你使用 **Raspbian Buster** 在 K3s 集群上安装 Rancher，请按照[这些步骤](https://rancher.com/docs/k3s/latest/en/advanced/#enabling-legacy-iptables-on-raspbian-buster)切换到旧版 iptables。 + +如果你使用 Alpine Linux 的 K3s 集群上安装 Rancher，请按照[这些步骤](https://rancher.com/docs/k3s/latest/en/advanced/#additional-preparation-for-alpine-linux-setup)进行其他设置。 + +### RKE 要求 + +RKE 需要 Docker 容器运行时。支持的 Docker 版本请参见 [Rancher 支持矩阵](https://www.suse.com/suse-rancher/support-matrix/all-supported-versions) + +有关详细信息，请参阅[安装 Docker](install-docker.md)。 + +## 硬件要求 + +本节描述安装 Rancher Server 的节点的 CPU、内存和磁盘要求。硬件要求根据你的 Rancher 部署规模而定。 + +### 实际考虑 + +Rancher 的硬件占用空间取决于许多因素，包括： + +- 托管的基础设施规模 (例如： 节点数量，集群数量)。 +- 所需访问控制规则的复杂性（例如：RoleBinding 对象计数）。 +- 工作负载数量 (例如： Kubernetes 部署，Fleet 部署)。 +- 使用模式 (例如：主动使用的功能集合，使用频率，并发用户数量). + +由于存在许多可能随时间变化的影响因素，因此此处列出的要求为适合大多数用例的起点。 然而，你的用例可能有不同的要求。 若你需要对于特定场景的咨询，请[联系 Rancher](<(https://rancher.com/contact/)>) 以获得进一步指导。 + +特别指出，本页面中的要求基于以下假设的环境提出，包括： + +- 每种类型的 Kubernetes 资源数量小于 60,000 个。 +- 每个节点最多 120 个 Pod。 +- 上游（本地）集群中最多 200 个 CRD。 +- 下游集群中最多 100 个 CRD。 +- 最多 50 个 Fleet 部署。 + +更多的数量也是能够达到的，但需要更高的硬件要求。 如果你有超过 20,000 个相同类型的资源，通过 Rancher UI 加载整个列表的时间可能需要几秒钟。 + +:::note Evolution: + +Rancher 的代码库不断发展，用例不断变化，Rancher 积累的经验也在不断增长。 + +随着指导方针的准确性不断的提高并且变得更加具体，硬件要求也会发生变化。 + +如果你发现你的 Rancher 部署不再符合列出的建议，请[联系 Rancher](https://rancher.com/contact/) 进行重新评估。 + +::: + +### RKE2 Kubernetes + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，生产环境下的高可用安装最少需要 3 个节点。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | +| -------- | ------------ | ------------ | ----- | ----- | +| 小 | 150 | 1500 | 4 | 16 GB | +| 中 | 300 | 3000 | 8 | 32 GB | +| 大 (\*) | 500 | 5000 | 16 | 64 GB | +| 更大 (†) | (†) | (†) | (†) | (†) | + +(\*)： 大规模的部署需要你[遵循最佳实践](../../../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md)以获得足够的性能。 + +(†)： 通过特别的硬件建议和调整能够实现更大的部署规模。 你可以[联系 Rancher](https://rancher.com/contact/) 进行定制评估。 + +有关 RKE2 一般要求的更多详细信息，请参见 [RKE2 文档](https://docs.rke2.io/install/requirements)。 + +### K3s Kubernetes + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，生产环境下的高可用安装最少需要 3 个节点。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | 外部数据库(\*) | +| --------- | ------------ | ------------ | ----- | ----- | -------------------------- | +| Small | 150 | 1500 | 4 | 16 GB | 2 vCPUs, 8 GB + 1000 IOPS | +| Medium | 300 | 3000 | 8 | 32 GB | 4 vCPUs, 16 GB + 2000 IOPS | +| Large (†) | 500 | 5000 | 16 | 64 GB | 8 vCPUs, 32 GB + 4000 IOPS | + +(\*)：外部数据库是指将 K3s 集群数据存储在[专用的外部主机](https://docs.k3s.io/datastore)上。 这是可选的。 具体要求取决于使用的外部数据库。 + +(†)：大规模的部署需要你[遵循最佳实践](../../../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md)以获得足够的性能。 + +有关 K3s 一般要求的更多详细信息，请参见 [K3s 文档](https://docs.k3s.io/installation/requirements)。 + +### 托管 Kubernetes + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，生产环境下的高可用安装最少需要 3 个节点。 + +这些要求适用于托管 Kubernetes 集群，例如 Amazon Elastic Kubernetes Service (EKS)、Azure Kubernetes Service (AKS) 或 Google Kubernetes Engine (GKE)。 它们不适用于 Rancher SaaS 解决方案，例如 [Rancher Prime Hosted](https://www.rancher.com/products/rancher)。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | +| -------- | ------------ | ------------ | ----- | ----- | +| 小 | 150 | 1500 | 4 | 16 GB | +| 中 | 300 | 3000 | 8 | 32 GB | +| 大 (\*) | 500 | 5000 | 16 | 64 GB | + +(\*)：大规模的部署需要你[遵循最佳实践](../../../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md)以获得足够的性能。 + +### RKE + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，生产环境下的高可用安装最少需要 3 个节点。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | +| -------- | ------------ | ------------ | ----- | ----- | +| 小 | 150 | 1500 | 4 | 16 GB | +| 中 | 300 | 3000 | 8 | 32 GB | +| 大 (\*) | 500 | 5000 | 16 | 64 GB | + +(\*)： 大规模的部署需要你[遵循最佳实践](../../../reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md)以获得足够的性能。 + +有关 RKE 一般要求的更多详细信息，请参见 [RKE 文档](https://rke.docs.rancher.com/os)。 + +### Docker + +下面的表格列出了[上游集群](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中每个节点最小的 CPU 和内存要求。 + +请注意，在 Docker 中安装 Rancher 仅适用于开发或测试目的。不建议在生产环境中使用。 + +| 部署规模 | 最大集群数量 | 最大节点数量 | vCPUs | 内存 | +| -------- | ------------ | ------------ | ----- | ---- | +| 小 | 5 | 50 | 1 | 4 GB | +| 中 | 15 | 200 | 2 | 8 GB | + +## Ingress + +安装 Rancher 的 Kubernetes 集群中的每个节点都应该运行一个 Ingress。 + +Ingress 需要部署为 DaemonSet 以确保负载均衡器能成功把流量转发到各个节点。 + +如果是 RKE，RKE2 和 K3s 安装，你不需要手动安装 Ingress，因为它是默认安装的。 + +对于托管的 Kubernetes 集群（EKS、GKE、AKS），你需要设置 Ingress。 + +- **Amazon EKS**：[在 Amazon EKS 上安装 Rancher 以及如何安装 Ingress 以访问 Rancher Server](../install-upgrade-on-a-kubernetes-cluster/rancher-on-amazon-eks.md)。 +- **AKS**：[使用 Azure Kubernetes 服务安装 Rancher 以及如何安装 Ingress 以访问 Rancher Server](../install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md)。 +- **GKE**：[使用 GKE 安装 Rancher 以及如何安装 Ingress 以访问 Rancher Server](../install-upgrade-on-a-kubernetes-cluster/rancher-on-gke.md)。 + +## 磁盘 + +etcd 在集群中的性能决定了 Rancher 的性能。因此，为了获得最佳速度，我们建议使用 SSD 磁盘来支持 Rancher 管理的 Kubernetes 集群。在云提供商上，你还需使用能获得最大 IOPS 的最小大小。在较大的集群中，请考虑使用专用存储设备存储 etcd 数据和 wal 目录。 + +## 网络要求 + +本节描述了安装 Rancher Server 的节点的网络要求。 + +:::caution + +如果包含 Rancher 的服务器带有 `X-Frame-Options=DENY` 标头，在升级旧版 UI 之后，Rancher UI 中的某些页面可能无法渲染。这是因为某些旧版页面在新 UI 中是以 iFrames 模式嵌入的。 + +::: + +### 节点 IP 地址 + +无论你是在单个节点还是高可用集群上安装 Rancher，每个节点都应配置一个静态 IP。如果使用 DHCP，则每个节点都应该有一个 DHCP 预留，以确保节点分配到相同的 IP 地址。 + +### 端口要求 + +为了确保能正常运行，Rancher 需要在 Rancher 节点和下游 Kubernetes 集群节点上开放一些端口。不同集群类型的 Rancher 和下游集群的所有必要端口，请参见[端口要求](port-requirements.md)。 + +## Dockershim 支持 + +有关 Dockershim 支持的详情，请参见[此页面](dockershim.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-requirements/port-requirements.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-requirements/port-requirements.md index 7dcdc48683c..ce924e173f9 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-requirements/port-requirements.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/installation-requirements/port-requirements.md @@ -3,6 +3,10 @@ title: 端口要求 description: 了解 Rancher 正常运行所需的端口要求，包括 Rancher 节点和下游 Kubernetes 集群节点 --- + + + + import PortsIaasNodes from '@site/src/components/PortsIaasNodes' import PortsCustomNodes from '@site/src/components/PortsCustomNodes' import PortsImportedHosted from '@site/src/components/PortsImportedHosted' @@ -47,22 +51,22 @@ K3s server 需要开放端口 6443 才能供节点访问。
Rancher Server 节点的入站规则
-| 协议 | 端口 | 源 | 描述 | -|-----|-----|----------------|---| -| TCP | 80 | 执行外部 SSL 终止的负载均衡器/代理 | 使用外部 SSL 终止时的 Rancher UI/API | -| TCP | 443 |
  • Server 节点
  • Agent 节点
  • 托管/注册的 Kubernetes
  • 任何需要使用 Rancher UI 或 API 的源
| Rancher Agent，Rancher UI/API，kubectl | -| TCP | 6443 | K3s Server 节点 | Kubernetes API | -| UDP | 8472 | K3s Server 和 Agent 节点 | 仅 Flannel VXLAN 需要 | -| TCP | 10250 | K3s Server 和 Agent 节点 | kubelet | +| 协议 | 端口 | 源 | 描述 | +| ---- | ----- | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | +| TCP | 80 | 执行外部 SSL 终止的负载均衡器/代理 | 使用外部 SSL 终止时的 Rancher UI/API | +| TCP | 443 |
  • Server 节点
  • Agent 节点
  • 托管/注册的 Kubernetes
  • 任何需要使用 Rancher UI 或 API 的源
| Rancher Agent，Rancher UI/API，kubectl | +| TCP | 6443 | K3s Server 节点 | Kubernetes API | +| UDP | 8472 | K3s Server 和 Agent 节点 | 仅 Flannel VXLAN 需要 | +| TCP | 10250 | K3s Server 和 Agent 节点 | kubelet |
Rancher 节点的出站规则
-| 协议 | 端口 | 目标 | 描述 | -| -------- | ---- | -------------------------------------------------------- | --------------------------------------------- | -| TCP | 22 | 使用 Node Driver 创建的节点的任何节点 IP | 使用 Node Driver SSH 配置节点 | -| TCP | 443 | git.rancher.io | Rancher catalog | -| TCP | 2376 | 使用 Node Driver 创建的节点的任何节点 IP | Docker Machine 使用的 Docker daemon TLS 端口 | -| TCP | 6443 | 托管/导入的 Kubernetes API | Kubernetes API Server | +| 协议 | 端口 | 目标 | 描述 | +| ---- | ---- | ---------------------------------------- | -------------------------------------------- | +| TCP | 22 | 使用 Node Driver 创建的节点的任何节点 IP | 使用 Node Driver SSH 配置节点 | +| TCP | 443 | git.rancher.io | Rancher catalog | +| TCP | 2376 | 使用 Node Driver 创建的节点的任何节点 IP | Docker Machine 使用的 Docker daemon TLS 端口 | +| TCP | 6443 | 托管/导入的 Kubernetes API | Kubernetes API Server |
@@ -73,44 +77,42 @@ K3s server 需要开放端口 6443 才能供节点访问。 通常情况下，Rancher 安装在三个 RKE 节点上，这些节点都有 etcd、controlplane 和 worker 角色。 - - 下表描述了 Rancher 节点之间流量的端口要求：
Rancher 节点的流量规则
-| 协议 | 端口 | 描述 | -|-----|-----|----------------| -| TCP | 443 | Rancher Agents | -| TCP | 2379 | etcd 客户端请求 | -| TCP | 2380 | etcd 对等通信 | -| TCP | 6443 | Kubernetes apiserver | -| TCP | 8443 | NGINX Ingress 的验证 Webhook | -| UDP | 8472 | Canal/Flannel VXLAN 覆盖网络 | -| TCP | 9099 | Canal/Flannel livenessProbe/readinessProbe | -| TCP | 10250 | Metrics Server 与所有节点的通信 | -| TCP | 10254 | Ingress controller livenessProbe/readinessProbe | +| 协议 | 端口 | 描述 | +| ---- | ----- | ----------------------------------------------- | +| TCP | 443 | Rancher Agents | +| TCP | 2379 | etcd 客户端请求 | +| TCP | 2380 | etcd 对等通信 | +| TCP | 6443 | Kubernetes apiserver | +| TCP | 8443 | NGINX Ingress 的验证 Webhook | +| UDP | 8472 | Canal/Flannel VXLAN 覆盖网络 | +| TCP | 9099 | Canal/Flannel livenessProbe/readinessProbe | +| TCP | 10250 | Metrics Server 与所有节点的通信 | +| TCP | 10254 | Ingress controller livenessProbe/readinessProbe | 下表描述了入站和出站流量的端口要求：
Rancher 节点的入站规则
-| 协议 | 端口 | 源 | 描述 | -|-----|-----|----------------|---| -| TCP | 22 | RKE CLI | RKE 通过 SSH 配置节点 | -| TCP | 80 | 负载均衡器/反向代理 | 到 Rancher UI/API 的 HTTP 流量 | -| TCP | 443 |
  • 负载均衡器/反向代理
  • 所有集群节点和其他 API/UI 客户端的 IP
| 到 Rancher UI/API 的 HTTPS 流量 | -| TCP | 6443 | Kubernetes API 客户端 | 到 Kubernetes API 的 HTTPS 流量 | +| 协议 | 端口 | 源 | 描述 | +| ---- | ---- | ----------------------------------------------------------------------------------- | ------------------------------- | +| TCP | 22 | RKE CLI | RKE 通过 SSH 配置节点 | +| TCP | 80 | 负载均衡器/反向代理 | 到 Rancher UI/API 的 HTTP 流量 | +| TCP | 443 |
  • 负载均衡器/反向代理
  • 所有集群节点和其他 API/UI 客户端的 IP
| 到 Rancher UI/API 的 HTTPS 流量 | +| TCP | 6443 | Kubernetes API 客户端 | 到 Kubernetes API 的 HTTPS 流量 |
Rancher 节点的出站规则
-| 协议 | 端口 | 目标 | 描述 | -|-----|-----|----------------|---| -| TCP | 443 | git.rancher.io | Rancher catalog | -| TCP | 22 | 使用 Node Driver 创建的任何节点 | Node Driver 通过 SSH 配置节点 | -| TCP | 2376 | 使用 Node Driver 创建的任何节点 | Node Driver 使用的 Docker daemon TLS 端口 | -| TCP | 6443 | 托管/导入的 Kubernetes API | Kubernetes API Server | -| TCP | 提供商依赖 | 托管集群中 Kubernetes API 端点的端口 | Kubernetes API | +| 协议 | 端口 | 目标 | 描述 | +| ---- | ---------- | ------------------------------------ | ----------------------------------------- | +| TCP | 443 | git.rancher.io | Rancher catalog | +| TCP | 22 | 使用 Node Driver 创建的任何节点 | Node Driver 通过 SSH 配置节点 | +| TCP | 2376 | 使用 Node Driver 创建的任何节点 | Node Driver 使用的 Docker daemon TLS 端口 | +| TCP | 6443 | 托管/导入的 Kubernetes API | Kubernetes API Server | +| TCP | 提供商依赖 | 托管集群中 Kubernetes API 端点的端口 | Kubernetes API |
@@ -133,20 +135,21 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点
RKE2 Server 节点的入站规则
-| 协议 | 端口 | 源 | 描述 | -|-----|-----|----------------|---| -| TCP | 9345 | RKE2 Server 和 Agent 节点 | 节点注册。需要在所有 Server 节点上将端口开放给集群中的所有其他节点。 | -| TCP | 6443 | RKE2 Agent 节点 | Kubernetes API | -| UDP | 8472 | RKE2 Server 和 Agent 节点 | 仅 Flannel VXLAN 需要 | -| TCP | 10250 | RKE2 Server 和 Agent 节点 | kubelet | -| TCP | 2379 | RKE2 Server 节点 | etcd 客户端端口 | -| TCP | 2380 | RKE2 Server 节点 | etcd 对等端口 | -| TCP | 30000-32767 | RKE2 Server 和 Agent 节点 | NodePort 端口范围。可以使用 TCP 或 UDP。 | -| TCP | 5473 | Calico-node pod 连接到 typha pod | 使用 Calico 部署时需要 | -| HTTP | 80 | 执行外部 SSL 终止的负载均衡器/代理 | 使用外部 SSL 终止时的 Rancher UI/API | -| HTTPS | 443 |
  • 托管/注册的 Kubernetes
  • 任何需要使用 Rancher UI 或 API 的源
| Rancher Agent，Rancher UI/API，kubectl。如果负载均衡器执行 TLS 终止，则不需要。 | +| 协议 | 端口 | 源 | 描述 | +| ----- | ----------- | ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------- | +| TCP | 9345 | RKE2 Server 和 Agent 节点 | 节点注册。需要在所有 Server 节点上将端口开放给集群中的所有其他节点。 | +| TCP | 6443 | RKE2 Agent 节点 | Kubernetes API | +| UDP | 8472 | RKE2 Server 和 Agent 节点 | 仅 Flannel VXLAN 需要 | +| TCP | 10250 | RKE2 Server 和 Agent 节点 | kubelet | +| TCP | 2379 | RKE2 Server 节点 | etcd 客户端端口 | +| TCP | 2380 | RKE2 Server 节点 | etcd 对等端口 | +| TCP | 30000-32767 | RKE2 Server 和 Agent 节点 | NodePort 端口范围。可以使用 TCP 或 UDP。 | +| TCP | 5473 | Calico-node pod 连接到 typha pod | 使用 Calico 部署时需要 | +| HTTP | 80 | 执行外部 SSL 终止的负载均衡器/代理 | 使用外部 SSL 终止时的 Rancher UI/API | +| HTTPS | 443 |
  • 托管/注册的 Kubernetes
  • 任何需要使用 Rancher UI 或 API 的源
| Rancher Agent，Rancher UI/API，kubectl。如果负载均衡器执行 TLS 终止，则不需要。 | 所有出站流量通常都是允许的。 +
### Docker 安装的 Rancher Server 的端口 @@ -158,19 +161,19 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点
Rancher 节点的入站规则
-| 协议 | 端口 | 源 | 描述 | -|-----|-----|----------------|---| -| TCP | 80 | 执行外部 SSL 终止的负载均衡器/代理 | 使用外部 SSL 终止时的 Rancher UI/API | -| TCP | 443 |
  • 托管/注册的 Kubernetes
  • 任何需要使用 Rancher UI 或 API 的源
| Rancher Agent，Rancher UI/API，kubectl | +| 协议 | 端口 | 源 | 描述 | +| ---- | ---- | ------------------------------------------------------------------------------------ | -------------------------------------- | +| TCP | 80 | 执行外部 SSL 终止的负载均衡器/代理 | 使用外部 SSL 终止时的 Rancher UI/API | +| TCP | 443 |
  • 托管/注册的 Kubernetes
  • 任何需要使用 Rancher UI 或 API 的源
| Rancher Agent，Rancher UI/API，kubectl |
Rancher 节点的出站规则
-| 协议 | 端口 | 源 | 描述 | -|-----|-----|----------------|---| -| TCP | 22 | 使用 Node Driver 创建的节点的任何节点 IP | 使用 Node Driver SSH 配置节点 | -| TCP | 443 | git.rancher.io | Rancher catalog | -| TCP | 2376 | 使用 Node Driver 创建的节点的任何节点 IP | Docker Machine 使用的 Docker daemon TLS 端口 | -| TCP | 6443 | 托管/导入的 Kubernetes API | Kubernetes API Server | +| 协议 | 端口 | 源 | 描述 | +| ---- | ---- | ---------------------------------------- | -------------------------------------------- | +| TCP | 22 | 使用 Node Driver 创建的节点的任何节点 IP | 使用 Node Driver SSH 配置节点 | +| TCP | 443 | git.rancher.io | Rancher catalog | +| TCP | 2376 | 使用 Node Driver 创建的节点的任何节点 IP | Docker Machine 使用的 Docker daemon TLS 端口 | +| TCP | 6443 | 托管/导入的 Kubernetes API | Kubernetes API Server |
@@ -178,9 +181,9 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点 下游 Kubernetes 集群用于运行你的应用和服务。本节介绍了哪些端口需要在下游集群的节点上打开，以便 Rancher 能够与它们进行通信。 -不同的下游集群的启动方式有不同的端口要求。下面的每个标签都列出了不同[集群类型](../../../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)所需打开的端口。 +不同的下游集群的启动方式有不同的端口要求。下面的每个标签都列出了不同[集群类型](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)所需打开的端口。 -下图描述了为每个[集群类型](../../../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)打开的端口。 +下图描述了为每个[集群类型](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)打开的端口。
Rancher 管理面板的端口要求
@@ -196,13 +199,12 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点 有关 Harvester 端口要求的更多信息，请参阅[此处](../../../integrations-in-rancher/harvester.md#端口要求)。 - ### Rancher 使用节点池启动 Kubernetes 集群的端口
单击展开 -下表描述了节点在[云提供商](../../../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md)中创建的情况下，[Rancher 启动 Kubernetes](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md) 的端口要求。 +下表描述了节点在[云提供商](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)中创建的情况下，[Rancher 启动 Kubernetes](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) 的端口要求。 :::note @@ -219,7 +221,7 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点
单击展开 -下表描述了使用[自定义节点](../../../pages-for-subheaders/use-existing-nodes.md)的情况下，[Rancher 启动 Kubernetes](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md) 的端口要求。 +下表描述了使用[自定义节点](../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)的情况下，[Rancher 启动 Kubernetes](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) 的端口要求。 @@ -230,7 +232,7 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点
单击展开 -下表描述了[托管集群](../../../pages-for-subheaders/set-up-clusters-from-hosted-kubernetes-providers.md)的端口要求。 +下表描述了[托管集群](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md)的端口要求。 @@ -253,18 +255,17 @@ RKE2 server 需要开放端口 6443 和 9345 才能供集群中的其他节点
- ## 其他端口注意事项 ### 常用端口 无论集群是什么类型，常用端口通常在你的 Kubernetes 节点上打开。 -import CommonPortsTable from '../../../shared-files/_common-ports-table.md'; +import CommonPortsTable from '../../../shared-files/\_common-ports-table.md'; ----- +--- ### 本地节点流量 @@ -282,21 +283,21 @@ import CommonPortsTable from '../../../shared-files/_common-ports-table.md'; 当你使用 [AWS EC2 Node Driver](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-amazon-ec2-cluster.md) 在 Rancher 中配置集群节点时，你可以让 Rancher 创建一个名为 `rancher-nodes` 的安全组。以下规则会自动添加到该安全组中。 -| 类型 | 协议 | 端口范围 | 源/目标 | 规则类型 | -|-----------------|:--------:|:-----------:|------------------------|:---------:| -| SSH | TCP | 22 | 0.0.0.0/0 | 入站 | -| HTTP | TCP | 80 | 0.0.0.0/0 | 入站 | -| 自定义 TCP 规则 | TCP | 443 | 0.0.0.0/0 | 入站 | -| 自定义 TCP 规则 | TCP | 2376 | 0.0.0.0/0 | 入站 | -| 自定义 TCP 规则 | TCP | 2379-2380 | sg-xxx (rancher-nodes) | 入站 | -| 自定义 UDP 规则 | UDP | 4789 | sg-xxx (rancher-nodes) | 入站 | -| 自定义 TCP 规则 | TCP | 6443 | 0.0.0.0/0 | 入站 | -| 自定义 UDP 规则 | UDP | 8472 | sg-xxx (rancher-nodes) | 入站 | -| 自定义 TCP 规则 | TCP | 10250-10252 | sg-xxx (rancher-nodes) | 入站 | -| 自定义 TCP 规则 | TCP | 10256 | sg-xxx (rancher-nodes) | 入站 | -| 自定义 TCP 规则 | TCP | 30000-32767 | 0.0.0.0/0 | 入站 | -| 自定义 UDP 规则 | UDP | 30000-32767 | 0.0.0.0/0 | 入站 | -| 所有流量 | 全部 | 全部 | 0.0.0.0/0 | 出站 | +| 类型 | 协议 | 端口范围 | 源/目标 | 规则类型 | +| --------------- | :--: | :---------: | ---------------------- | :------: | +| SSH | TCP | 22 | 0.0.0.0/0 | 入站 | +| HTTP | TCP | 80 | 0.0.0.0/0 | 入站 | +| 自定义 TCP 规则 | TCP | 443 | 0.0.0.0/0 | 入站 | +| 自定义 TCP 规则 | TCP | 2376 | 0.0.0.0/0 | 入站 | +| 自定义 TCP 规则 | TCP | 2379-2380 | sg-xxx (rancher-nodes) | 入站 | +| 自定义 UDP 规则 | UDP | 4789 | sg-xxx (rancher-nodes) | 入站 | +| 自定义 TCP 规则 | TCP | 6443 | 0.0.0.0/0 | 入站 | +| 自定义 UDP 规则 | UDP | 8472 | sg-xxx (rancher-nodes) | 入站 | +| 自定义 TCP 规则 | TCP | 10250-10252 | sg-xxx (rancher-nodes) | 入站 | +| 自定义 TCP 规则 | TCP | 10256 | sg-xxx (rancher-nodes) | 入站 | +| 自定义 TCP 规则 | TCP | 30000-32767 | 0.0.0.0/0 | 入站 | +| 自定义 UDP 规则 | UDP | 30000-32767 | 0.0.0.0/0 | 入站 | +| 所有流量 | 全部 | 全部 | 0.0.0.0/0 | 出站 | ### 打开 SUSE Linux 端口 @@ -307,12 +308,14 @@ SUSE Linux 可能有一个防火墙，默认情况下会阻止所有端口。要 1. SSH 进入实例。 1. 以文本模式启动 YaST： + ``` sudo yast2 ``` 1. 导航到**安全和用户** > **防火墙** > **区域：公共** > **端口**。要在界面内导航，请参照[说明](https://doc.opensuse.org/documentation/leap/reference/html/book-reference/cha-yast-text.html#sec-yast-cli-navigate)。 1. 要打开所需的端口，把它们输入到 **TCP 端口** 和 **UDP 端口** 字段。在这个例子中，端口 9796 和 10250 也被打开，用于监控。由此产生的字段应类似于以下内容： + ```yaml TCP Ports 22, 80, 443, 2376, 2379, 2380, 6443, 9099, 9796, 10250, 10254, 30000-32767 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md new file mode 100644 index 00000000000..0e00dcde72d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md @@ -0,0 +1,36 @@ +--- +title: 离线 Helm CLI 安装 +--- + + + + + +本文介绍如何使用 Helm CLI 在离线环境中安装 Rancher Server。离线环境可以是 Rancher Server 离线安装、防火墙后面或代理后面。 + +Rancher 安装在 RKE Kubernetes 集群、K3s Kubernetes 集群，或单个 Docker 容器上对应的安装步骤会有所不同。 + +如需了解各个安装方式的更多信息，请参见[本页](../../installation-and-upgrade.md)。 + +在安装指导中，我们为不同的安装选项提供对应的 _选项卡_ 。 + +:::note 重要提示： + +如果你按照 Docker 安装指南安装 Rancher，你将没有把 Docker 安装转换为 Kubernetes 安装的升级途径。 + +::: + +## 安装概要 + +1. [设置基础设施和私有镜像仓库](infrastructure-private-registry.md) +2. [收集镜像到私有镜像仓库](publish-images.md) +3. [设置 Kubernetes 集群（如果你使用 Docker 安装，请跳过此步骤）](install-kubernetes.md) +4. [安装 Rancher](install-rancher-ha.md) + +## 升级 + +如需在离线环境中使用 Helm CLI 升级 Rancher，请按照[升级步骤](../../install-upgrade-on-a-kubernetes-cluster/upgrades.md)进行操作。 + +### 后续操作 + +[准备节点](infrastructure-private-registry.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/infrastructure-private-registry.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/infrastructure-private-registry.md index e3e7be1a627..171ec478b1d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/infrastructure-private-registry.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/infrastructure-private-registry.md @@ -1,12 +1,16 @@ --- -title: '1. 设置基础设施和私有镜像仓库' +title: "1. 设置基础设施和私有镜像仓库" --- + + + + 本文介绍如何在离线环境中，为 Rancher Management server 配置底层基础设施。你还将设置 Rancher 节点中必须可用的私有容器镜像仓库。 离线环境是 Rancher Server 离线安装或安装在防火墙后面的环境。 -Rancher 安装在 K3s Kubernetes 集群、RKE Kubernetes 集群还是单个 Docker 容器上对应的基础设施设置会有所不同。如需了解各个安装方式的更多信息，请参见[本页](../../../../pages-for-subheaders/installation-and-upgrade.md)。 +Rancher 安装在 K3s Kubernetes 集群、RKE Kubernetes 集群还是单个 Docker 容器上对应的基础设施设置会有所不同。如需了解各个安装方式的更多信息，请参见[本页](../../installation-and-upgrade.md)。 Rancher 可以安装在任何 Kubernetes 集群上。为了阅读方便，我们在下文中仍提供了 RKE 和 K3s Kubernetes 基础设施教程。 @@ -25,7 +29,7 @@ Rancher 可以安装在任何 Kubernetes 集群上。为了阅读方便，我们 这些主机会断开互联网链接，但需要能与你的私有镜像仓库连接。 -请确保你的节点满足[操作系统，容器运行时，硬件和网络](../../../../pages-for-subheaders/installation-requirements.md)的常规要求。 +请确保你的节点满足[操作系统，容器运行时，硬件和网络](../../installation-requirements/installation-requirements.md)的常规要求。 如需获取配置 Linux 节点的示例，请参见[在 Amazon EC2 中配置节点](../../../../how-to-guides/new-user-guides/infrastructure-setup/nodes-in-amazon-ec2.md)的教程。 @@ -35,9 +39,9 @@ K3s 与其他 Kubernetes 发行版不同，在于其支持使用 etcd 以外的 对于 K3s 高可用安装，你需要配置以下的其中一个数据库： -* [PostgreSQL](https://www.postgresql.org/)（10.7 和 11.5 已验证） -* [MySQL](https://www.mysql.com/)（5.7 已验证） -* [etcd](https://etcd.io/)（3.3.15 已验证） +- [PostgreSQL](https://www.postgresql.org/)（10.7 和 11.5 已验证） +- [MySQL](https://www.mysql.com/)（5.7 已验证） +- [etcd](https://etcd.io/)（3.3.15 已验证） 在安装 Kubernetes 时，你需要传入 K3s 连接数据库的详细信息。 @@ -86,9 +90,9 @@ Rancher 支持使用私有镜像仓库进行离线安装。你必须有自己的 如果你需要创建私有镜像仓库，请参阅相应运行时的文档： -* [Containerd](https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration). - * [Nerdctl 命令和镜像仓库托管服务](https://github.com/containerd/nerdctl/blob/main/docs/registry.md) -* [Docker](https://docs.docker.com/registry/deploying/). +- [Containerd](https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration). + - [Nerdctl 命令和镜像仓库托管服务](https://github.com/containerd/nerdctl/blob/main/docs/registry.md) +- [Docker](https://docs.docker.com/registry/deploying/). @@ -157,9 +161,9 @@ Rancher 支持使用安全的私有镜像仓库进行离线安装。你必须有 如果你需要创建私有镜像仓库，请参阅相应运行时的文档： -* [Containerd](https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration). - * [Nerdctl 命令和镜像仓库托管服务](https://github.com/containerd/nerdctl/blob/main/docs/registry.md) -* [Docker](https://docs.docker.com/registry/deploying/). +- [Containerd](https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration). + - [Nerdctl 命令和镜像仓库托管服务](https://github.com/containerd/nerdctl/blob/main/docs/registry.md) +- [Docker](https://docs.docker.com/registry/deploying/). @@ -190,4 +194,5 @@ Rancher 支持使用私有镜像仓库在堡垒服务器中进行离线安装。 ### 后续操作 + [收集镜像并发布到你的私有镜像仓库](publish-images.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha.md index 0aa220051f7..d4ac43ceaf7 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha.md @@ -2,6 +2,10 @@ title: 4. 安装 Rancher --- + + + + 本文介绍如何在高可用 Kubernetes 安装的离线环境部署 Rancher。离线环境可以是 Rancher Server 离线安装、防火墙后面或代理后面。 ### Rancher 特权访问 @@ -23,26 +27,29 @@ title: 4. 安装 Rancher 1. 如果你还没有安装 `helm`，请在可访问互联网的工作站上进行本地安装。注意：参考 [Helm 版本要求](../../resources/helm-version-requirements.md)选择 Helm 版本来安装 Rancher。 2. 执行 `helm repo add` 命令，以添加包含安装 Rancher 的 Chart 的 Helm Chart 仓库。有关如何选择仓库，以及哪个仓库最适合你的用例，请参见[选择 Rancher 版本](../../resources/choose-a-rancher-version.md)。 + - Latest：建议用于试用最新功能 - ``` - helm repo add rancher-latest https://releases.rancher.com/server-charts/latest - ``` + ``` + helm repo add rancher-latest https://releases.rancher.com/server-charts/latest + ``` - Stable：建议用于生产环境 - ``` - helm repo add rancher-stable https://releases.rancher.com/server-charts/stable - ``` + ``` + helm repo add rancher-stable https://releases.rancher.com/server-charts/stable + ``` - Alpha：即将发布的实验性预览。 - ``` - helm repo add rancher-alpha https://releases.rancher.com/server-charts/alpha - ``` - 注意：不支持升级到 Alpha 版、从 Alpha 版升级或在 Alpha 版之间升级。 + ``` + helm repo add rancher-alpha https://releases.rancher.com/server-charts/alpha + ``` + 注意：不支持升级到 Alpha 版、从 Alpha 版升级或在 Alpha 版之间升级。 3. 获取最新的 Rancher Chart。此操作将获取 Chart 并将其作为 `.tgz` 文件保存在当前目录中。 + ```plain helm fetch rancher-/rancher ``` 如需下载特定的 Rancher 版本，你可以用 Helm `--version` 参数指定版本，如下： + ```plain helm fetch rancher-stable/rancher --version=v2.4.8 ``` @@ -59,20 +66,20 @@ Rancher Server 默认设计为安全的，并且需要 SSL/TLS 配置。 ::: -| 配置 | Chart 选项 | 描述 | 是否需要 cert-manager | -| ------------------------------------------ | ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------- | -| Rancher 生成的自签名证书 | `ingress.tls.source=rancher` | 使用 Rancher 生成的 CA 签发的自签名证书。此项是**默认选项**。在渲染 Helm 模板的时候不需要传递此项。 | 是 | -| 你已有的证书 | `ingress.tls.source=secret` | 通过创建 Kubernetes 密文使用你自己的证书文件。
在渲染 Rancher Helm 模板时必须传递此选项。 | 否 | +| 配置 | Chart 选项 | 描述 | 是否需要 cert-manager | +| ------------------------ | ---------------------------- | --------------------------------------------------------------------------------------------------- | --------------------- | +| Rancher 生成的自签名证书 | `ingress.tls.source=rancher` | 使用 Rancher 生成的 CA 签发的自签名证书。此项是**默认选项**。在渲染 Helm 模板的时候不需要传递此项。 | 是 | +| 你已有的证书 | `ingress.tls.source=secret` | 通过创建 Kubernetes 密文使用你自己的证书文件。
在渲染 Rancher Helm 模板时必须传递此选项。 | 否 | ### 离线安装的 Helm Chart 选项 在配置 Rancher Helm 模板时，Helm Chart 中有几个专为离线安装设计的选项，如下表： -| Chart 选项 | Chart 值 | 描述 | -| ----------------------- | -------------------------------- | ---- | -| `certmanager.version` | `` | 根据运行的 cert-manager 版本配置适当的 Rancher TLS 颁发者。 | -| `systemDefaultRegistry` | `` | 将 Rancher Server 配置成在配置集群时，始终从私有镜像仓库中拉取镜像。 | -| `useBundledSystemChart` | `true` | 配置 Rancher Server 使用打包的 Helm System Chart 副本。[system charts](https://github.com/rancher/system-charts) 仓库包含所有 Monitoring，Logging，告警和全局 DNS 等功能所需的应用商店项目。这些 [Helm Chart](https://github.com/rancher/system-charts) 位于 GitHub 中。但是由于你处在离线环境，因此使用 Rancher 内置的 Chart 会比设置 Git mirror 容易得多。 | +| Chart 选项 | Chart 值 | 描述 | +| ----------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `certmanager.version` | `` | 根据运行的 cert-manager 版本配置适当的 Rancher TLS 颁发者。 | +| `systemDefaultRegistry` | `` | 将 Rancher Server 配置成在配置集群时，始终从私有镜像仓库中拉取镜像。 | +| `useBundledSystemChart` | `true` | 配置 Rancher Server 使用打包的 Helm System Chart 副本。[system charts](https://github.com/rancher/system-charts) 仓库包含所有 Monitoring，Logging，告警和全局 DNS 等功能所需的应用商店项目。这些 [Helm Chart](https://github.com/rancher/system-charts) 位于 GitHub 中。但是由于你处在离线环境，因此使用 Rancher 内置的 Chart 会比设置 Git mirror 容易得多。 | ### 3. 获取 Cert-Manager Chart @@ -102,14 +109,15 @@ helm repo update 从 [Helm Chart 仓库](https://artifacthub.io/packages/helm/cert-manager/cert-manager)中获取最新可用的 cert-manager Chart： ```plain -helm fetch jetstack/cert-manager --version v1.11.0 +helm fetch jetstack/cert-manager ``` ##### 3. 检索 Cert-Manager CRD 为 cert-manager 下载所需的 CRD 文件： + ```plain -curl -L -o cert-manager-crd.yaml https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml +curl -L -o cert-manager-crd.yaml https://github.com/cert-manager/cert-manager/releases/download//cert-manager.crds.yaml ``` ### 4. 安装 Rancher @@ -146,7 +154,7 @@ curl -L -o cert-manager-crd.yaml https://github.com/cert-manager/cert-manager/re 3. 安装 cert-manager。 ```plain - helm install cert-manager ./cert-manager-v1.11.0.tgz \ + helm install cert-manager ./cert-manager-.tgz \ --namespace cert-manager \ --set image.repository=/quay.io/jetstack/cert-manager-controller \ --set webhook.image.repository=/quay.io/jetstack/cert-manager-webhook \ @@ -157,6 +165,7 @@ curl -L -o cert-manager-crd.yaml https://github.com/cert-manager/cert-manager/re
##### 2. 安装 Rancher + 首先，参见[添加 TLS 密文](../../resources/add-tls-secrets.md)发布证书文件，以便 Rancher 和 Ingress Controller 可以使用它们。 然后，使用 kubectl 为 Rancher 创建命名空间： @@ -169,12 +178,12 @@ kubectl create namespace cattle-system 对于 Kubernetes v1.25 或更高版本，使用 Rancher v2.7.2-v2.7.4 时，将 `global.cattle.psp.enabled` 设置为 `false`。对于 Rancher v2.7.5 及更高版本来说，这不是必需的，但你仍然可以手动设置该选项。 -| 占位符 | 描述 | -------------|------------- -| `` | 输出压缩包的版本号。 | -| `` | 指向负载均衡器的 DNS 名称。 | -| `` | 你的私有镜像仓库的 DNS 名称。 | -| `` | 在 K8s 集群上运行的 cert-manager 版本。 | +| 占位符 | 描述 | +| -------------------------------- | --------------------------------------- | +| `` | 输出压缩包的版本号。 | +| `` | 指向负载均衡器的 DNS 名称。 | +| `` | 你的私有镜像仓库的 DNS 名称。 | +| `` | 在 K8s 集群上运行的 cert-manager 版本。 | ```plain helm install rancher ./rancher-.tgz \ @@ -200,10 +209,10 @@ kubectl create namespace cattle-system 对于 Kubernetes v1.25 或更高版本，使用 Rancher v2.7.2-v2.7.4 时，将 `global.cattle.psp.enabled` 设置为 `false`。对于 Rancher v2.7.5 及更高版本来说，这不是必需的，但你仍然可以手动设置该选项。 -| 占位符 | 描述 | -| -------------------------------- | ----------------------------------------------- | -| `` | 输出压缩包的版本号。 | -| `` | 指向负载均衡器的 DNS 名称。 | +| 占位符 | 描述 | +| -------------------------------- | ----------------------------- | +| `` | 输出压缩包的版本号。 | +| `` | 指向负载均衡器的 DNS 名称。 | | `` | 你的私有镜像仓库的 DNS 名称。 | ```plain @@ -229,7 +238,6 @@ kubectl create namespace cattle-system --set useBundledSystemChart=true # 使用打包的 Rancher System Chart ``` - 安装已完成。 :::caution diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md index c64f7132d1d..e67f6ad708d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/publish-images.md @@ -1,14 +1,18 @@ --- -title: '2. 收集镜像并发布到私有仓库' +title: "2. 收集镜像并发布到私有仓库" --- + + + + 本文介绍如何配置私有镜像仓库，以便在安装 Rancher 时，Rancher 可以从此私有镜像仓库中拉取所需的镜像。 -默认情况下，所有用于[配置 Kubernetes 集群](../../../../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)或启动 Rancher 中的工具（如监控，流水线，告警等）的镜像都是从 Docker Hub 中拉取的。在 Rancher 的离线安装中，你需要一个私有仓库，该仓库位于你的 Rancher Server 中某个可访问的位置。然后，你可加载该存有所有镜像的镜像仓库。 +默认情况下，所有用于[配置 Kubernetes 集群](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)或启动 Rancher 中的工具（如监控，流水线，告警等）的镜像都是从 Docker Hub 中拉取的。在 Rancher 的离线安装中，你需要一个私有仓库，该仓库位于你的 Rancher Server 中某个可访问的位置。然后，你可加载该存有所有镜像的镜像仓库。 使用 Docker 安装 Rancher，和把 Rancher 安装到 Kubernetes 集群，其对应的推送镜像到私有镜像仓库步骤是一样的。 -你使用 Rancher 配置的下游集群是否有运行 Windows 的节点，决定了本文涉及的步骤。我们提供的推送镜像到私有镜像仓库步骤，是基于假设 Rancher 仅配置运行 Linux 节点的下游 Kubernetes 集群的。但是，如果你计划[在下游 Kubernetes 集群中使用 Windows 节点](../../../../pages-for-subheaders/use-windows-clusters.md)，我们有单独的文档来介绍如何为需要的镜像提供支持。 +你使用 Rancher 配置的下游集群是否有运行 Windows 的节点，决定了本文涉及的步骤。我们提供的推送镜像到私有镜像仓库步骤，是基于假设 Rancher 仅配置运行 Linux 节点的下游 Kubernetes 集群的。但是，如果你计划[在下游 Kubernetes 集群中使用 Windows 节点](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)，我们有单独的文档来介绍如何为需要的镜像提供支持。 :::note 先决条件： @@ -40,11 +44,11 @@ title: '2. 收集镜像并发布到私有仓库' 2. 从你所需版本的 **Assets** 处下载以下文件，这些文件是在离线环境中安装 Rancher 所必须的： -| Release 文件 | 描述 | -| ---------------- | -------------- | -| `rancher-images.txt` | 此文件包含安装 Rancher、配置集群和用户 Rancher 工具所需的镜像。 | +| Release 文件 | 描述 | +| ------------------------ | -------------------------------------------------------------------------------------------------------------------- | +| `rancher-images.txt` | 此文件包含安装 Rancher、配置集群和用户 Rancher 工具所需的镜像。 | | `rancher-save-images.sh` | 该脚本从 Docker Hub 中拉取在文件 `rancher-images.txt` 中记录的所有镜像，并把这些镜像保存为 `rancher-images.tar.gz`。 | -| `rancher-load-images.sh` | 该脚本从 `rancher-images.tar.gz` 文件中加载镜像，并把镜像推送到你的私有镜像仓库。 | +| `rancher-load-images.sh` | 该脚本从 `rancher-images.tar.gz` 文件中加载镜像，并把镜像推送到你的私有镜像仓库。 | ### 2. 收集 cert-manager 镜像 @@ -67,7 +71,7 @@ title: '2. 收集镜像并发布到私有仓库' ```plain helm repo add jetstack https://charts.jetstack.io helm repo update - helm fetch jetstack/cert-manager --version v1.11.0 + helm fetch jetstack/cert-manager helm template ./cert-manager-.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt ``` @@ -80,6 +84,7 @@ title: '2. 收集镜像并发布到私有仓库' ### 3. 将镜像保存到你的工作站中 1. 为 `rancher-save-images.sh` 文件添加可执行权限： + ``` chmod +x rancher-save-images.sh ``` @@ -99,15 +104,19 @@ title: '2. 收集镜像并发布到私有仓库' `rancher-images.txt` 需要位于工作站中运行 `rancher-load-images.sh` 脚本的同一目录中。`rancher-images.tar.gz` 也需要位于同一目录中。 1. 登录到你的私有镜像仓库： + ```plain docker login ``` + 1. 为 `rancher-load-images.sh` 添加可执行权限： + ``` chmod +x rancher-load-images.sh ``` 1. 使用 `rancher-load-images.sh` 脚本来提取，标记和推送 `rancher-images.txt` 及 `rancher-images.tar.gz` 到你的私有镜像仓库： + ```plain ./rancher-load-images.sh --image-list ./rancher-images.txt --registry ``` @@ -142,11 +151,11 @@ chmod +x rancher-load-images.sh 2. 从你所需版本的 **Assets** 处下载以下文件： -| Release 文件 | 描述 | -|----------------------------|------------------| -| `rancher-windows-images.txt` | 此文件包含配置 Windows 集群所需的 Windows 镜像。 | -| `rancher-save-images.ps1` | 该脚本从 Docker Hub 中拉取在文件 `rancher-windows-images.txt` 中记录的所有镜像，并把这些镜像保存为 `rancher-windows-images.tar.gz`。 | -| `rancher-load-images.ps1` | 该脚本从 `rancher-windows-images.tar.gz` 文件中加载镜像，并把镜像推送到你的私有镜像仓库。 | +| Release 文件 | 描述 | +| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `rancher-windows-images.txt` | 此文件包含配置 Windows 集群所需的 Windows 镜像。 | +| `rancher-save-images.ps1` | 该脚本从 Docker Hub 中拉取在文件 `rancher-windows-images.txt` 中记录的所有镜像，并把这些镜像保存为 `rancher-windows-images.tar.gz`。 | +| `rancher-load-images.ps1` | 该脚本从 `rancher-windows-images.tar.gz` 文件中加载镜像，并把镜像推送到你的私有镜像仓库。 | @@ -155,6 +164,7 @@ chmod +x rancher-load-images.sh 1. 在 `powershell` 中，进入上一步下载的文件所在的目录。 1. 运行 `rancher-save-images.ps1` 以创建包含所有所需镜像的压缩包： + ```plain ./rancher-save-images.ps1 ``` @@ -187,6 +197,7 @@ chmod +x rancher-load-images.sh `rancher-windows-images.txt` 需要位于工作站中运行 `rancher-load-images.ps1` 脚本的同一目录中。`rancher-windows-images.tar.gz` 也需要位于同一目录中。 1. 使用 `powershell` 登录到你的私有镜像仓库： + ```plain docker login ``` @@ -221,12 +232,12 @@ Linux 镜像需要在 Linux 主机上收集和推送，但是你必须先将 Win 2. 从你所需版本的 **Assets** 处下载以下文件： -| Release 文件 | 描述 | -|----------------------------| -------------------------- | -| `rancher-images.txt` | 此文件包含安装 Rancher、配置集群和用户 Rancher 工具所需的镜像。 | -| `rancher-windows-images.txt` | 此文件包含配置 Windows 集群所需的镜像。 | -| `rancher-save-images.sh` | 该脚本从 Docker Hub 中拉取在文件 `rancher-images.txt` 中记录的所有镜像，并把这些镜像保存为 `rancher-images.tar.gz`。 | -| `rancher-load-images.sh` | 该脚本从 `rancher-images.tar.gz` 文件中加载镜像，并把镜像推送到你的私有镜像仓库。 | +| Release 文件 | 描述 | +| ---------------------------- | -------------------------------------------------------------------------------------------------------------------- | +| `rancher-images.txt` | 此文件包含安装 Rancher、配置集群和用户 Rancher 工具所需的镜像。 | +| `rancher-windows-images.txt` | 此文件包含配置 Windows 集群所需的镜像。 | +| `rancher-save-images.sh` | 该脚本从 Docker Hub 中拉取在文件 `rancher-images.txt` 中记录的所有镜像，并把这些镜像保存为 `rancher-images.tar.gz`。 | +| `rancher-load-images.sh` | 该脚本从 `rancher-images.tar.gz` 文件中加载镜像，并把镜像推送到你的私有镜像仓库。 | @@ -234,7 +245,6 @@ Linux 镜像需要在 Linux 主机上收集和推送，但是你必须先将 Win **在 Kubernetes 安装中，如果你使用的是 Rancher 默认的自签名 TLS 证书**，则必须将 [`cert-manager`](https://artifacthub.io/packages/helm/cert-manager/cert-manager) 镜像添加到 `rancher-images.txt` 文件中。如果你使用自己的证书，则可跳过此步骤。 - 1. 获取最新的 `cert-manager` Helm Chart，并解析模板以获取镜像的详情信息： :::note @@ -246,7 +256,7 @@ Linux 镜像需要在 Linux 主机上收集和推送，但是你必须先将 Win ```plain helm repo add jetstack https://charts.jetstack.io helm repo update - helm fetch jetstack/cert-manager --version v1.11.0 + helm fetch jetstack/cert-manager helm template ./cert-manager-.tgz | awk '$1 ~ /image:/ {print $2}' | sed s/\"//g >> ./rancher-images.txt ``` @@ -260,6 +270,7 @@ Linux 镜像需要在 Linux 主机上收集和推送，但是你必须先将 Win ### 3. 将镜像保存到你的工作站中 1. 为 `rancher-save-images.sh` 文件添加可执行权限： + ``` chmod +x rancher-save-images.sh ``` @@ -280,11 +291,13 @@ Linux 镜像需要在 Linux 主机上收集和推送，但是你必须先将 Win 镜像列表，即 `rancher-images.txt` 或 `rancher-windows-images.txt` 需要位于工作站中运行 `rancher-load-images.sh` 脚本的同一目录中。`rancher-images.tar.gz` 也需要位于同一目录中。 1. 登录到你的私有镜像仓库： + ```plain docker login ``` 1. 为 `rancher-load-images.sh` 添加可执行权限： + ``` chmod +x rancher-load-images.sh ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/other-installation-methods.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/other-installation-methods.md new file mode 100644 index 00000000000..ca46a69c974 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/other-installation-methods.md @@ -0,0 +1,23 @@ +--- +title: 其他安装方式 +--- + + + + + +### 离线安装 + +按照[以下步骤](air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)在离线环境中安装 Rancher Server。 + +离线环境可以是 Rancher Server 离线安装、防火墙后面或代理后面。 + +### Docker 安装 + +[单节点 Docker 安装](rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md)适用于想要测试 Rancher 的用户。你无需使用 Helm 在 Kubernetes 集群上运行 Rancher，你可以使用 `docker run` 命令，把 Rancher Server 组件安装到单个节点上。 + +Docker 安装仅用于开发和测试环境。 + +由于只有一个节点和一个 Docker 容器，因此，如果该节点发生故障，由于其他节点上没有可用的 etcd 数据副本，你将丢失 Rancher Server 的所有数据。 + +Rancher backup operator 可将 Rancher 从单个 Docker 容器迁移到高可用 Kubernetes 集群上。详情请参见[把 Rancher 迁移到新集群](../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher.md index da7bd9fb728..3a587efa532 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/install-rancher.md @@ -2,10 +2,16 @@ title: 3. 安装 Rancher --- + + + + 在前文的操作后，你已经有了一个运行的 RKE 集群，现在可以在其中安装 Rancher 了。出于安全考虑，所有到 Rancher 的流量都必须使用 TLS 加密。在本教程中，你将使用 [cert-manager](https://cert-manager.io/)自动颁发自签名证书。在实际使用情况下，你可使用 Let's Encrypt 或自己的证书。 ### 安装 Helm CLI + + 在具有 kubeconfig 的主机上安装 [Helm](https://helm.sh/docs/intro/install/) CLI 以访问 Kubernetes 集群： ``` @@ -31,7 +37,7 @@ kubectl create namespace cert-manager 安装 cert-manager 的 CustomResourceDefinitions： ``` -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download//cert-manager.crds.yaml ``` 使用 Helm 安装 cert-manager。请注意，cert-manager 还需要你配置代理，以防它需要与 Let's Encrypt 或其他外部证书颁发商进行通信： @@ -44,7 +50,7 @@ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/ ``` helm upgrade --install cert-manager jetstack/cert-manager \ - --namespace cert-manager --version v1.11.0 \ + --namespace cert-manager \ --set http_proxy=http://${proxy_host} \ --set https_proxy=http://${proxy_host} \ --set no_proxy=127.0.0.0/8\\,10.0.0.0/8\\,cattle-system.svc\\,172.16.0.0/12\\,192.168.0.0/16\\,.svc\\,.cluster.local diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md new file mode 100644 index 00000000000..969171dd320 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/rancher-behind-an-http-proxy.md @@ -0,0 +1,17 @@ +--- +title: 使用 HTTP 代理安装 Rancher +--- + + + + + +很多企业本地运行的服务器或虚拟机不能直接访问互联网，但是出于安全考虑，他们必须通过 HTTP(S) 代理连接到外部服务。本教程将分步介绍如何在这样的环境中进行高可用的 Rancher 安装。 + +另外，用户也可以在没有任何互联网访问的情况下离线设置 Rancher。详情请参见 [Rancher 官方文档](../air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)。 + +## 安装概要 + +1. [配置基础设施](set-up-infrastructure.md) +2. [配置 Kubernetes 集群](install-kubernetes.md) +3. [安装 Rancher](install-rancher.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure.md index 58414b8ed1d..c5aefdf29ca 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-behind-an-http-proxy/set-up-infrastructure.md @@ -1,7 +1,11 @@ --- -title: '1. 配置基础设施' +title: "1. 配置基础设施" --- + + + + 在本节中，你将为 Rancher Management Server 配置底层基础设施，并使其通过 HTTP 代理访问互联网。 如需在高可用 RKE 集群中安装 Rancher Management Server，我们建议配置以下基础设施： @@ -22,7 +26,7 @@ title: '1. 配置基础设施' 这些主机将通过 HTTP 代理连接到互联网。 -请确保你的节点满足[操作系统，容器运行时，硬件和网络](../../../../pages-for-subheaders/installation-requirements.md)的常规要求。 +请确保你的节点满足[操作系统，容器运行时，硬件和网络](../../installation-requirements/installation-requirements.md)的常规要求。 如需获取配置 Linux 节点的示例，请参见[在 Amazon EC2 中配置节点](../../../../how-to-guides/new-user-guides/infrastructure-setup/nodes-in-amazon-ec2.md)的教程。 @@ -59,6 +63,6 @@ title: '1. 配置基础设施' 有关设置 DNS 记录以将域流量转发到 Amazon ELB 负载均衡器的指南，请参见 [AWS 官方文档](https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-to-elb-load-balancer)。 - ### 后续操作 + [配置 Kubernetes 集群](install-kubernetes.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md index 6e1ac8ee365..590d36c0ded 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md @@ -2,19 +2,25 @@ title: 证书故障排除 --- + + + + + + ### 如何确定我的证书格式是否为 PEM？ 你可以通过以下特征识别 PEM 格式： - 文件开始的标头： - ``` - -----BEGIN CERTIFICATE----- - ``` + ``` + -----BEGIN CERTIFICATE----- + ``` - 表头后跟一长串字符。 - 文件结束的页脚： - ``` - -----END CERTIFICATE----- - ``` + ``` + -----END CERTIFICATE----- + ``` PEM 证书示例： diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md new file mode 100644 index 00000000000..944dee55c46 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md @@ -0,0 +1,214 @@ +--- +title: 使用 Docker 将 Rancher 安装到单个节点中 +description: 在开发和测试环境中，你可以使用 Docker 安装。在单个 Linux 主机上安装 Docker，然后使用一个 Docker 容器部署 Rancher。 +--- + + + + + + + +Rancher 可以通过运行单个 Docker 容器进行安装。 + +在这种安装方案中，你需要将 Docker 安装到单个 Linux 主机，然后使用单个 Docker 容器将 Rancher 部署到主机中。 + +:::note 想要使用外部负载均衡器？ + +请参阅[使用外部负载均衡器的 Docker 安装](../../../../how-to-guides/advanced-user-guides/configure-layer-7-nginx-load-balancer.md)。 + +::: + +Rancher 的 Docker 安装仅推荐用于开发和测试环境中。Rancher 版本决定了能否将 Rancher 迁移到高可用集群。 + +Rancher backup operator 可将 Rancher 从单个 Docker 容器迁移到高可用 Kubernetes 集群上。详情请参见[把 Rancher 迁移到新集群](../../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)。 + +## Rancher 特权访问 + +当 Rancher Server 部署在 Docker 容器中时，容器内会安装一个本地 Kubernetes 集群供 Rancher 使用。为 Rancher 的很多功能都是以 deployment 的方式运行的，而在容器内运行容器是需要特权模式的，因此你需要在安装 Rancher 时添加 `--privileged` 选项。 + +## 操作系统，Docker，硬件和网络要求 + +请确保你的节点满足常规的[安装要求](../../installation-requirements/installation-requirements.md)。 + +## 1. 配置 Linux 主机 + +按照[要求](../../installation-requirements/installation-requirements.md)配置一个 Linux 主机，用于运行 Rancher Server。 + +## 2. 选择一个 SSL 选项并安装 Rancher + +出于安全考虑，使用 Rancher 时请使用 SSL（Secure Sockets Layer）。SSL 保护所有 Rancher 网络通信（如登录和与集群交互）的安全。 + +:::tip 你是否需要： + +- 使用代理。参见 [HTTP 代理配置](../../../../reference-guides/single-node-rancher-in-docker/http-proxy-configuration.md)。 +- 配置自定义 CA 根证书以访问服务。参见[自定义 CA 根证书](../../../../reference-guides/single-node-rancher-in-docker/advanced-options.md#自定义-ca-证书)。 +- 完成离线安装。参见 [离线：Docker 安装](../air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)。 +- 记录所有 Rancher API 的事务。参加 [API 审计](../../../../reference-guides/single-node-rancher-in-docker/advanced-options.md#api-审计日志)。 + +::: + +选择以下的选项之一： + +- [选项 A：使用 Rancher 生成的默认自签名证书](#选项-a使用-rancher-生成的默认自签名证书) +- [选项 B：使用你自己的证书 - 自签名](#选项-b使用你自己的证书---自签名) +- [选项 C：使用你自己的证书 - 可信 CA 签名的证书](#选项-c使用你自己的证书---可信-ca-签名的证书) +- [选项 D：Let's Encrypt 证书](#选项-dlets-encrypt-证书) +- [选项 E：Localhost 隧道，不使用证书](#选项-elocalhost-隧道不使用证书) + +### 选项 A：使用 Rancher 生成的默认自签名证书 + +如果你在不考虑身份验证的开发或测试环境中安装 Rancher，可以使用 Rancher 生成的自签名证书安装 Rancher。这种安装方式避免了自己生成证书的麻烦。 + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + --privileged \ + rancher/rancher:latest +``` + +### 选项 B：使用你自己的证书 - 自签名 + +在你团队访问 Rancher Server 的开发或测试环境中，创建一个用于你的安装的自签名证书，以便团队验证他们对实例的连接。 + +:::note 先决条件： + +使用 [OpenSSL](https://www.openssl.org/) 或其他方法创建自签名证书。 + +- 证书文件的格式必须是 PEM。 +- 在你的证书文件中，包括链中的所有中间证书。你需要对你的证书进行排序，把你的证书放在最前面，后面跟着中间证书。如需查看示例，请参见[证书故障排除](../getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md)。 + +::: + +创建证书后，运行以下 Docker 命令以安装 Rancher。使用 `-v` 标志并提供证书的路径，以将证书挂载到容器中。 + +| 占位符 | 描述 | +| ------------------- | -------------------------- | +| `` | 包含证书文件的目录的路径。 | +| `` | 完整证书链的路径。 | +| `` | 证书私钥的路径。 | +| `` | CA 证书的路径。 | + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + -v //:/etc/rancher/ssl/cert.pem \ + -v //:/etc/rancher/ssl/key.pem \ + -v //:/etc/rancher/ssl/cacerts.pem \ + --privileged \ + rancher/rancher:latest +``` + +### 选项 C：使用你自己的证书 - 可信 CA 签名的证书 + +在公开暴露应用的生产环境中，请使用由可信 CA 签名的证书，以避免用户收到证书安全警告。 + +不建议将 Docker 安装用于生产环境。这些说明仅适用于测试和开发。 + +:::note 先决条件： + +- 证书文件的格式必须是 PEM。 +- 在你的证书文件中，包括可信 CA 提供的所有中间证书。你需要对你的证书进行排序，把你的证书放在最前面，后面跟着中间证书。如需查看示例，请参见[证书故障排除](../getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/certificate-troubleshooting.md)。 + +::: + +获取证书后，运行以下 Docker 命令。 + +- 使用 `-v` 标志并提供证书的路径，以将证书挂载到容器中。因为你的证书是由可信的 CA 签名的，因此你不需要安装额外的 CA 证书文件。 +- 使用 `--no-cacerts` 作为容器的参数，以禁用 Rancher 生成的默认 CA 证书。 + +| 占位符 | 描述 | +| ------------------- | -------------------------- | +| `` | 包含证书文件的目录的路径。 | +| `` | 完整证书链的路径。 | +| `` | 证书私钥的路径。 | + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + -v //:/etc/rancher/ssl/cert.pem \ + -v //:/etc/rancher/ssl/key.pem \ + --privileged \ + rancher/rancher:latest \ + --no-cacerts +``` + +### 选项 D：Let's Encrypt 证书 + +:::caution + +Let's Encrypt 对新证书请求有频率限制。因此，请限制创建或销毁容器的频率。详情请参见 [Let's Encrypt 官方文档 - 频率限制](https://letsencrypt.org/docs/rate-limits/)。 + +::: + +你也可以在生产环境中使用 [Let's Encrypt](https://letsencrypt.org/) 证书。Let's Encrypt 使用 HTTP-01 质询来验证你对域名的控制权。如果要确认你对该域名有控制权，你可将用于访问 Rancher 的主机名（例如 `rancher.mydomain.com`）指向运行的主机的 IP。你可通过在 DNS 中创建 A 记录，以将主机名绑定到 IP 地址。 + +不建议将 Docker 安装用于生产环境。这些说明仅适用于测试和开发。 + +:::note 先决条件： + +- Let's Encrypt 是联网服务。因此，在内网和离线环境中不能使用。 +- 在 DNS 中创建一条记录，将 Linux 主机 IP 地址绑定到要用于访问 Rancher 的主机名（例如，`rancher.mydomain.com`）。 +- 在 Linux 主机上打开 `TCP/80` 端口。Let's Encrypt 的 HTTP-01 质询可以来自任何源 IP 地址，因此端口 `TCP/80` 必须开放开所有 IP 地址。 + +::: + +满足先决条件后，你可以运行以下命令使用 Let's Encrypt 证书安装 Rancher。 + +| 占位符 | 描述 | +| ----------------- | ------------ | +| `` | 你的域名地址 | + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + --privileged \ + rancher/rancher:latest \ + --acme-domain +``` + +### 选项 E：Localhost 隧道，不使用证书 + +如果你在开发或测试环境中安装 Rancher，且环境中有运行的 localhost 隧道解决方案（如 [ngrok](https://ngrok.com/)），不要生成证书。此安装选项不需要证书。 + +- 使用 `--no-cacerts` 作为参数，以禁用 Rancher 生成的默认 CA 证书。 + +登录到你的主机，然后运行以下命令： + +``` +docker run -d --restart=unless-stopped \ + -p 80:80 -p 443:443 \ + --privileged \ + rancher/rancher:latest \ + --no-cacerts +``` + +## 高级选项 + +使用 Docker 将 Rancher 安装到单个节点时，有如下几个可开启的高级选项： + +- 自定义 CA 证书 +- API 审计日志 +- TLS 设置 +- 离线环境 +- 持久化数据 +- 在同一个节点中运行 `rancher/rancher` 和 `rancher/rancher-agent` + +详情请参见[本页](../../../../reference-guides/single-node-rancher-in-docker/advanced-options.md)。 + +## 故障排除 + +如需了解常见问题及故障排除提示，请参见[本页](certificate-troubleshooting.md)。 + +## 后续操作 + +- **推荐**：检查单节点[备份](../../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-docker-installed-rancher.md)和[恢复](../../../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-docker-installed-rancher.md)。你可能暂时没有需要备份的数据，但是我们建议你在常规使用 Rancher 后创建备份。 +- 创建 Kubernetes 集群：[配置 Kubernetes 集群](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/roll-back-docker-installed-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/roll-back-docker-installed-rancher.md index 32fbf2f4254..88b37bdf85f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/roll-back-docker-installed-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/roll-back-docker-installed-rancher.md @@ -2,6 +2,12 @@ title: 回滚 Docker 安装的 Rancher --- + + + + + + 如果 Rancher 升级没有成功完成，你需要回滚到你在 [Docker 升级](upgrade-docker-installed-rancher.md)之前使用的 Rancher 设置。回滚可以恢复： - 先前版本的 Rancher。 @@ -21,12 +27,13 @@ docker pull rancher/rancher: 终端 docker ps 命令，显示如何找到 <PRIOR_RANCHER_VERSION> 和 <RANCHER_CONTAINER_NAME>![占位符参考](/img/placeholder-ref-2.png) -| 占位符 | 示例 | 描述 | -| -------------------------- | -------------------------- | ------------------------------------------------------- | -| `` | `v2.0.5` | 升级前使用的 rancher/rancher 镜像。 | -| `` | `festive_mestorf` | 你的 Rancher 容器的名称。 | -| `` | `v2.0.5` | 备份对应的 Rancher 版本。 | -| `` | `9-27-18` | 数据容器或备份的创建日期。 | +| 占位符 | 示例 | 描述 | +| -------------------------- | ----------------- | ----------------------------------- | +| `` | `v2.0.5` | 升级前使用的 rancher/rancher 镜像。 | +| `` | `festive_mestorf` | 你的 Rancher 容器的名称。 | +| `` | `v2.0.5` | 备份对应的 Rancher 版本。 | +| `` | `9-27-18` | 数据容器或备份的创建日期。 | +
可以通过远程连接登录到 Rancher Server 所在的主机并输入命令 `docker ps` 以查看正在运行的容器，从而获得 `` 和 `` 。你还可以运行 `docker ps -a` 命令查看停止了的容器。在创建备份期间，你随时可以运行这些命令来获得帮助。 @@ -56,6 +63,7 @@ docker pull rancher/rancher: ``` docker stop ``` + 你可输入 `docker ps`获取 Rancher 容器的名称。 1. 将你在 [Docker 升级](upgrade-docker-installed-rancher.md)时创建的备份压缩包移动到 Rancher Server。切换到你将其移动到的目录。输入 `dir` 以确认它在该位置。 @@ -71,6 +79,7 @@ docker pull rancher/rancher: ``` 1. 将 `` 占位符指向数据容器，启动一个新的 Rancher Server 容器。 + ``` docker run -d --volumes-from rancher-data \ --restart=unless-stopped \ @@ -78,7 +87,8 @@ docker pull rancher/rancher: --privileged \ rancher/rancher: ``` - 特权访问是[必须](../../../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#rancher-特权访问)的。 + + 特权访问是[必须](rancher-on-a-single-node-with-docker.md#rancher-特权访问)的。 :::danger diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md index 68d23b34c33..705ce07983d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/other-installation-methods/rancher-on-a-single-node-with-docker/upgrade-docker-installed-rancher.md @@ -2,6 +2,10 @@ title: 升级 Docker 安装的 Rancher --- + + + + 本文介绍如何升级通过 Docker 安装的 Rancher Server。 :::caution @@ -13,7 +17,7 @@ title: 升级 Docker 安装的 Rancher ## 先决条件 - 在 Rancher 文档中**检查[已知升级问题](../../install-upgrade-on-a-kubernetes-cluster/upgrades.md#已知升级问题)**，了解升级 Rancher 时最需要注意的问题。你可以在 [GitHub](https://github.com/rancher/rancher/releases) 发布说明以及 [Rancher 论坛](https://forums.rancher.com/c/announcements/12)中找到每个 Rancher 版本的已知问题。不支持升级或升级到 [rancher-alpha 仓库](../../resources/choose-a-rancher-version.md#helm-chart-仓库)中的任何 Chart。 -- **[仅适用于离线安装](../../../../pages-for-subheaders/air-gapped-helm-cli-install.md)：为新的 Rancher Server 版本收集和推送镜像**。按照指南为你想要升级的目标 Rancher 版本[推送镜像到私有镜像仓库](../air-gapped-helm-cli-install/publish-images.md)。 +- **[仅适用于离线安装](../air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)：为新的 Rancher Server 版本收集和推送镜像**。按照指南为你想要升级的目标 Rancher 版本[推送镜像到私有镜像仓库](../air-gapped-helm-cli-install/publish-images.md)。 ## 占位符 @@ -41,12 +45,13 @@ docker ps ![占位符参考](/img/placeholder-ref.png) -| 占位符 | 示例 | 描述 | -| -------------------------- | -------------------------- | --------------------------------------------------------- | -| `` | `v2.1.3` | 首次安装拉取的 rancher/rancher 镜像。 | -| `` | `festive_mestorf` | 你的 Rancher 容器的名称。 | -| `` | `v2.1.3` | 你为其创建备份的 Rancher 版本。 | -| `` | `2018-12-19` | 数据容器或备份的创建日期。 | +| 占位符 | 示例 | 描述 | +| -------------------------- | ----------------- | ------------------------------------- | +| `` | `v2.1.3` | 首次安装拉取的 rancher/rancher 镜像。 | +| `` | `festive_mestorf` | 你的 Rancher 容器的名称。 | +| `` | `v2.1.3` | 你为其创建备份的 Rancher 版本。 | +| `` | `2018-12-19` | 数据容器或备份的创建日期。 | +
可以通过远程连接登录到 Rancher Server 所在的主机并输入命令 `docker ps` 以查看正在运行的容器，从而获得 `` 和 ``。你还可以运行 `docker ps -a` 命令查看停止了的容器。在创建备份期间，你随时可以运行这些命令来获得帮助。 @@ -54,6 +59,7 @@ docker ps ## 升级 在升级期间，你可以为当前 Rancher 容器创建数据的副本及备份，以确保可以在升级出现问题时可以进行回滚。然后，你可使用现有数据将新版本的 Rancher 部署到新容器中。 + ### 1. 创建 Rancher Server 容器的数据副本 1. 使用远程终端连接，登录到运行 Rancher Server 的节点。 @@ -75,6 +81,7 @@ docker ps 1. 从你刚刚创建的数据容器（rancher-data）中，创建一个备份 tar 包（rancher-data-backup-<RANCHER_VERSION>-<DATE>.tar.gz）。 如果升级期间出现问题，此压缩包可以用作回滚点。替换占位符来运行以下命令。 + ``` docker run --volumes-from rancher-data -v "$PWD:/backup" --rm busybox tar zcvf /backup/rancher-data-backup--.tar.gz /var/lib/rancher ``` @@ -94,8 +101,8 @@ docker ps 拉取你需要升级到的 Rancher 版本镜像。 -| 占位符 | 描述 | -------------|------------- +| 占位符 | 描述 | +| ----------------------- | ---------------------------------------------------------------------------------------------- | | `` | 你想要升级到的 [Rancher 版本](../../installation-references/helm-chart-options.md)的版本标签。 | ``` @@ -135,8 +142,8 @@ docker pull rancher/rancher: 如果你使用 Rancher 生成的自签名证书，则将 `--volumes-from rancher-data` 添加到你启动原始 Rancher Server 容器的命令中。 -| 占位符 | 描述 | -------------|------------- +| 占位符 | 描述 | +| ----------------------- | ---------------------------------------------------------------------------------------------- | | `` | 你想要升级到的 [Rancher 版本](../../installation-references/helm-chart-options.md)的版本标签。 | ``` @@ -164,12 +171,12 @@ docker run -d --volumes-from rancher-data \ ::: -| 占位符 | 描述 | -------------|------------- -| `` | 包含证书文件的目录的路径。 | -| `` | 完整证书链的路径。 | -| `` | 证书私钥的路径。 | -| `` | CA 证书的路径。 | +| 占位符 | 描述 | +| ----------------------- | ---------------------------------------------------------------------------------------------- | +| `` | 包含证书文件的目录的路径。 | +| `` | 完整证书链的路径。 | +| `` | 证书私钥的路径。 | +| `` | CA 证书的路径。 | | `` | 你想要升级到的 [Rancher 版本](../../installation-references/helm-chart-options.md)的版本标签。 | ``` @@ -200,11 +207,11 @@ docker run -d --volumes-from rancher-data \ ::: -| 占位符 | 描述 | -------------|------------- -| `` | 包含证书文件的目录的路径。 | -| `` | 完整证书链的路径。 | -| `` | 证书私钥的路径。 | +| 占位符 | 描述 | +| ----------------------- | ---------------------------------------------------------------------------------------------- | +| `` | 包含证书文件的目录的路径。 | +| `` | 完整证书链的路径。 | +| `` | 证书私钥的路径。 | | `` | 你想要升级到的 [Rancher 版本](../../installation-references/helm-chart-options.md)的版本标签。 | ``` @@ -219,6 +226,7 @@ docker run -d --volumes-from rancher-data \ ``` 特权访问是[必须](../../../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#rancher-特权访问)的。 +
#### 选项 D：Let's Encrypt 证书 @@ -241,10 +249,10 @@ Let's Encrypt 对新证书请求有频率限制。因此，请限制创建或销 ::: -| 占位符 | 描述 | -------------|------------- +| 占位符 | 描述 | +| ----------------------- | ---------------------------------------------------------------------------------------------- | | `` | 你想要升级到的 [Rancher 版本](../../installation-references/helm-chart-options.md)的版本标签。 | -| `` | 你最初使用的域名 | +| `` | 你最初使用的域名 | ``` docker run -d --volumes-from rancher-data \ @@ -273,10 +281,10 @@ docker run -d --volumes-from rancher-data \ 如果你使用 Rancher 生成的自签名证书，则将 `--volumes-from rancher-data` 添加到你启动原始 Rancher Server 容器的命令中。 -| 占位符 | 描述 | -------------|------------- -| `` | 私有镜像仓库的 URL 和端口。 | -| `` | 你想要升级到的 [Rancher 版本](../../installation-references/helm-chart-options.md)的版本标签。 | +| 占位符 | 描述 | +| -------------------------------- | ---------------------------------------------------------------------------------------------- | +| `` | 私有镜像仓库的 URL 和端口。 | +| `` | 你想要升级到的 [Rancher 版本](../../installation-references/helm-chart-options.md)的版本标签。 | ``` docker run -d --volumes-from rancher-data \ @@ -289,6 +297,7 @@ docker run -d --volumes-from rancher-data \ ``` 特权访问是[必须](../../../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#rancher-特权访问)的。 +
#### 选项 B：使用你自己的证书 - 自签名 @@ -304,14 +313,14 @@ docker run -d --volumes-from rancher-data \ ::: -| 占位符 | 描述 | -------------|------------- -| `` | 包含证书文件的目录的路径。 | -| `` | 完整证书链的路径。 | -| `` | 证书私钥的路径。 | -| `` | CA 证书的路径。 | -| `` | 私有镜像仓库的 URL 和端口。 | -| `` | 你想要升级到的 [Rancher 版本](../../installation-references/helm-chart-options.md)的版本标签。 | +| 占位符 | 描述 | +| -------------------------------- | ---------------------------------------------------------------------------------------------- | +| `` | 包含证书文件的目录的路径。 | +| `` | 完整证书链的路径。 | +| `` | 证书私钥的路径。 | +| `` | CA 证书的路径。 | +| `` | 私有镜像仓库的 URL 和端口。 | +| `` | 你想要升级到的 [Rancher 版本](../../installation-references/helm-chart-options.md)的版本标签。 | ``` docker run -d --restart=unless-stopped \ @@ -324,7 +333,9 @@ docker run -d --restart=unless-stopped \ --privileged \ /rancher/rancher: ``` + 特权访问是[必须](../../../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#rancher-特权访问)的。 +
#### 选项 C：使用你自己的证书 - 可信 CA 签名的证书 @@ -340,13 +351,13 @@ docker run -d --restart=unless-stopped \ ::: -| 占位符 | 描述 | -------------|------------- -| `` | 包含证书文件的目录的路径。 | -| `` | 完整证书链的路径。 | -| `` | 证书私钥的路径。 | -| `` | 私有镜像仓库的 URL 和端口。 | -| `` | 你想要升级到的 [Rancher 版本](../../installation-references/helm-chart-options.md)的版本标签。 | +| 占位符 | 描述 | +| -------------------------------- | ---------------------------------------------------------------------------------------------- | +| `` | 包含证书文件的目录的路径。 | +| `` | 完整证书链的路径。 | +| `` | 证书私钥的路径。 | +| `` | 私有镜像仓库的 URL 和端口。 | +| `` | 你想要升级到的 [Rancher 版本](../../installation-references/helm-chart-options.md)的版本标签。 | :::note @@ -366,7 +377,9 @@ docker run -d --volumes-from rancher-data \ --privileged /rancher/rancher: ``` + 特权访问是[必须](../../../../pages-for-subheaders/rancher-on-a-single-node-with-docker.md#rancher-特权访问)的。 + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/add-tls-secrets.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/add-tls-secrets.md index dc58f216be0..4a5fec73c31 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/add-tls-secrets.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/add-tls-secrets.md @@ -2,6 +2,10 @@ title: 添加 TLS 密文 --- + + + + 我们使用证书和密钥将 `cattle-system` 命名空间中的 `tls-rancher-ingress` 密文配置好后，Kubernetes 会为 Rancher 创建对象和服务。 将服务器证书和所需的所有中间证书合并到名为 `tls.crt`的文件中。将证书密钥复制到名为 `tls.key` 的文件中。 @@ -42,4 +46,4 @@ Rancher 启动时会检索配置的 `tls-ca` 密文。如果 Rancher 在运行 ## 更新私有 CA 证书 -按照[步骤](update-rancher-certificate.md)更新 [Rancher 高可用 Kubernetes 安装](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md)中的 Ingress，或从默认自签名证书切换到自定义证书。 \ No newline at end of file +按照[步骤](update-rancher-certificate.md)更新 [Rancher 高可用 Kubernetes 安装](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)中的 Ingress，或从默认自签名证书切换到自定义证书。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md index b168ecdcb05..d64aebb7751 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/choose-a-rancher-version.md @@ -2,6 +2,10 @@ title: 选择 Rancher 版本 --- + + + + 本节介绍如何选择 Rancher 版本。 在我们推荐用于生产环境的 Rancher 高可用安装中，Rancher Server 是通过 Kubernetes 集群上的 **Helm Chart** 安装的。请参见 [Helm 版本要求](helm-version-requirements.md)选择 Helm 版本来安装 Rancher。 @@ -11,7 +15,7 @@ title: 选择 Rancher 版本 -如果 Rancher Server 是[安装在 Kubernetes 集群上](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md)的，Rancher Server 的安装，升级和回滚中，都是使用 Kubernetes 集群上的 Helm Chart 来安装 Rancher 的。因此，在准备安装或升级 Rancher 高可用时，必须添加包含用于安装 Rancher 的 Chart 的 Helm Chart 仓库。 +如果 Rancher Server 是[安装在 Kubernetes 集群上](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)的，Rancher Server 的安装，升级和回滚中，都是使用 Kubernetes 集群上的 Helm Chart 来安装 Rancher 的。因此，在准备安装或升级 Rancher 高可用时，必须添加包含用于安装 Rancher 的 Chart 的 Helm Chart 仓库。 请参见 [Helm 版本要求](helm-version-requirements.md)选择 Helm 版本来安装 Rancher。 @@ -19,11 +23,11 @@ title: 选择 Rancher 版本 Rancher 提供几个可选的 Helm Chart 仓库供你选择。最新版或稳定版的 Helm Chart 仓库与用于 Docker 安装中的 Docker 标签对应。因此，`rancher-latest` 仓库包含所有标记为 `rancher/rancher:latest` 的 Rancher 版本 Chart。当 Rancher 版本升级到 `rancher/rancher:stable`，它会被添加到 `rancher-stable` 仓库中。 -| 类型 | 添加仓库的命令 | 仓库描述 | -| -------------- | ------------ | ----------------- | -| rancher-latest | `helm repo add rancher-latest https://releases.rancher.com/server-charts/latest` | 添加最新版本的 Rancher 的 Helm Chart 仓库。建议使用此仓库来测试新版本的 Rancher。 | -| rancher-stable | `helm repo add rancher-stable https://releases.rancher.com/server-charts/stable` | 添加较旧的，稳定的版本的 Rancher 的 Helm Chart 仓库。建议在生产环境中使用此仓库。 | -| rancher-alpha | `helm repo add rancher-alpha https://releases.rancher.com/server-charts/alpha` | 添加 alpha 版本的 Rancher 的 Helm Chart 仓库，以预览即将发布的版本。不建议在生产环境中使用这些版本。无论是什么仓库，均不支持 _升级_ 或 _升级到_ rancher-alpha 仓库中的任何 Chart。 | +| 类型 | 添加仓库的命令 | 仓库描述 | +| -------------- | -------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| rancher-latest | `helm repo add rancher-latest https://releases.rancher.com/server-charts/latest` | 添加最新版本的 Rancher 的 Helm Chart 仓库。建议使用此仓库来测试新版本的 Rancher。 | +| rancher-stable | `helm repo add rancher-stable https://releases.rancher.com/server-charts/stable` | 添加较旧的，稳定的版本的 Rancher 的 Helm Chart 仓库。建议在生产环境中使用此仓库。 | +| rancher-alpha | `helm repo add rancher-alpha https://releases.rancher.com/server-charts/alpha` | 添加 alpha 版本的 Rancher 的 Helm Chart 仓库，以预览即将发布的版本。不建议在生产环境中使用这些版本。无论是什么仓库，均不支持 _升级_ 或 _升级到_ rancher-alpha 仓库中的任何 Chart。 | 了解何时选择这些仓库，请参见[切换到不同 Helm Chart 仓库](#切换到不同-helm-chart-仓库)。 @@ -55,18 +59,18 @@ Rancher Helm Chart 版本与 Rancher 版本（即 `appVersion`）对应。添加 ::: - Latest：建议用于试用最新功能 - ``` - helm repo add rancher-latest https://releases.rancher.com/server-charts/latest - ``` + ``` + helm repo add rancher-latest https://releases.rancher.com/server-charts/latest + ``` - Stable：建议用于生产环境 - ``` - helm repo add rancher-stable https://releases.rancher.com/server-charts/stable - ``` + ``` + helm repo add rancher-stable https://releases.rancher.com/server-charts/stable + ``` - Alpha：即将发布的实验性预览。 - ``` - helm repo add rancher-alpha https://releases.rancher.com/server-charts/alpha - ``` - 注意：不支持升级到 Alpha 版、从 Alpha 版升级或在 Alpha 版之间升级。 + ``` + helm repo add rancher-alpha https://releases.rancher.com/server-charts/alpha + ``` + 注意：不支持升级到 Alpha 版、从 Alpha 版升级或在 Alpha 版之间升级。 1. 列出当前 Helm Chart 仓库。 @@ -101,10 +105,10 @@ Rancher Helm Chart 版本与 Rancher 版本（即 `appVersion`）对应。添加 Rancher Server 以 Docker 镜像的形式分发并附有标签。你可以在输入命令部署 Rancher 时指定标签。请记住，如果你指定了标签，但是没有指定版本（如 `latest` 或 `stable`），你必须显式拉取该镜像标签的新版本。否则，将使用缓存在主机上的镜像。 -| 标签 | 描述 | -| -------------------------- | ------ | -| `rancher/rancher:latest` | 最新的开发版本。这些版本通过了我们的 CI 自动化验证。不推荐在生产环境使用这些版本。 | -| `rancher/rancher:stable` | 最新的稳定版本。推荐将此标签用于生产环境。 | +| 标签 | 描述 | +| -------------------------- | -------------------------------------------------------------------------------------- | +| `rancher/rancher:latest` | 最新的开发版本。这些版本通过了我们的 CI 自动化验证。不推荐在生产环境使用这些版本。 | +| `rancher/rancher:stable` | 最新的稳定版本。推荐将此标签用于生产环境。 | | `rancher/rancher:` | 你可以使用以前版本中的标签来指定要安装的 Rancher 版本。访问 DockerHub 查看可用的版本。 | :::note diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/helm-version-requirements.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/helm-version-requirements.md index 273711a4de0..eab811d8fd2 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/helm-version-requirements.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/helm-version-requirements.md @@ -2,9 +2,15 @@ title: Helm 版本要求 --- + + + + 本文介绍 Helm 的要求。Helm 是用于把 Rancher 安装在高可用 Kubernetes 集群上的工具。 -> 我们已针对 Helm 3 更新了安装指南。如果你使用 Helm 2 进行安装，请参见 [Helm 2 迁移到 Helm 3 文档](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/)。[本文](/versioned_docs/version-2.0-2.4/pages-for-subheaders/helm2.md)提供了较早的使用 Helm 2 的 Rancher 高可用安装指南的副本。如果你如果无法升级到 Helm 3，可以使用这个说明安装。 +> 我们已针对 Helm 3 更新了安装指南。如果你使用 Helm 2 进行安装，请参见 [Helm 2 迁移到 Helm 3 文档](https://helm.sh/blog/migrate-from-helm-v2-to-helm-v3/)。[本文](/versioned_docs/version-2.0-2.4/getting-started/installation-and-upgrade/advanced-options/advanced-use-cases/helm2/helm2.md)提供了较早的使用 Helm 2 的 Rancher 高可用安装指南的副本。如果你如果无法升级到 Helm 3，可以使用这个说明安装。 + + - 如需安装或升级 Rancher 2.5，请使用 Helm 3.2.x 或更高版本。 - Kubernetes 1.16 要求 Helm 2.16.0 或更高版本。如果使用的是默认 Kubernetes 版本，请参见[发行说明](https://github.com/rancher/rke/releases)获取所使用的 RKE 版本。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/local-system-charts.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/local-system-charts.md index 4048c5b2566..b7f725496f6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/local-system-charts.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/local-system-charts.md @@ -2,6 +2,10 @@ title: 离线安装中设置本地 System Charts --- + + + + [Charts](https://github.com/rancher/charts) 仓库包含 Monitoring、Logging、告警和 Istio 等功能所需的所有 Helm 目录项。 在 Rancher 的离线安装中，你需要配置 Rancher 以使用 System Charts 的本地副本。本节介绍如何通过 CLI 标志使用本地 System Charts。 @@ -10,4 +14,4 @@ title: 离线安装中设置本地 System Charts `system-charts` 的一个本地副本已经打包到 `rancher/rancher` 容器中。为了在离线安装中使用这些功能，你需要使用额外的环境变量 `CATTLE_SYSTEM_CATALOG=bundled` 来运行 Rancher 安装命令，该环境变量告诉 Rancher 使用 Chart 的本地副本，而不是尝试从 GitHub 获取 Chart。 -带有 `system-charts` 的 Rancher 安装命令示例包含在 Docker 和 Helm 的[离线安装说明](../../../pages-for-subheaders/air-gapped-helm-cli-install.md)中。 +带有 `system-charts` 的 Rancher 安装命令示例包含在 Docker 和 Helm 的[离线安装说明](../other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)中。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/resources.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/resources.md new file mode 100644 index 00000000000..7267b559922 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/resources.md @@ -0,0 +1,29 @@ +--- +title: 资源 +--- + + + + + +### Docker 安装 + +[单节点 Docker 安装](../other-installation-methods/rancher-on-a-single-node-with-docker/rancher-on-a-single-node-with-docker.md)适用于想要测试 Rancher 的用户。你无需使用 Helm 在 Kubernetes 集群上运行 Rancher，你可以使用 `docker run` 命令，把 Rancher Server 组件安装到单个节点上。 + +由于只有一个节点和一个 Docker 容器，因此，如果该节点发生故障，由于其他节点上没有可用的 etcd 数据副本，你将丢失 Rancher Server 的所有数据。 + +### 离线安装 + +按照[以下步骤](../other-installation-methods/air-gapped-helm-cli-install/air-gapped-helm-cli-install.md)在离线环境中安装 Rancher Server。 + +离线环境可以是 Rancher Server 离线安装、防火墙后面或代理后面。 + +### 高级选项 + +安装 Rancher 时，有如下几个可开启的高级选项：每个安装指南中都提供了对应的选项。了解选项详情： + +- [自定义 CA 证书](custom-ca-root-certificates.md) +- [API 审计日志](../../../how-to-guides/advanced-user-guides/enable-api-audit-log.md) +- [TLS 设置](../installation-references/tls-settings.md) +- [etcd 配置](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md) +- [离线安装 Local System Chart](local-system-charts.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md index 3d11417cda6..8f920b8b32e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/update-rancher-certificate.md @@ -2,9 +2,13 @@ title: 更新 Rancher 证书 --- + + + + ## 更新私有 CA 证书 -按照以下步骤轮换[安装在 Kubernetes 集群上](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md)、由 Rancher 使用的 SSL 证书和私有 CA，或转用由私有 CA 签发的 SSL 证书。 +按照以下步骤轮换[安装在 Kubernetes 集群上](../install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)、由 Rancher 使用的 SSL 证书和私有 CA，或转用由私有 CA 签发的 SSL 证书。 步骤概述： @@ -69,6 +73,7 @@ kubectl -n cattle-system create secret generic tls-ca \ 在这种情况下，由于 `tls-ca` secret 在启动时由 Rancher pod 读取，因此你需要重新部署 Rancher pod。 你可以运行以下命令重新部署 Rancher pod： + ```bash kubectl rollout restart deploy/rancher -n cattle-system ``` @@ -82,24 +87,32 @@ kubectl rollout restart deploy/rancher -n cattle-system 以下步骤更新了 Rancher Chart 的 Helm 值，因此 Rancher pod 和 ingress 会使用在步骤 1 和 2 中创建的新私有 CA 证书。 1. 调整初始安装期间使用的值，将当前值存储为： + ```bash helm get values rancher -n cattle-system -o yaml > values.yaml ``` + 1. 检索当前部署的 Rancher Chart 的版本字符串： + ```bash helm ls -n cattle-system ``` + 1. 更新 `values.yaml` 文件中的当前 Helm 值以包含下方内容： + ```yaml ingress: tls: source: secret privateCA: true ``` + :::note 重要： 由于证书由私有 CA 签发，因此确保在 `values.yaml` 文件中设置了 [`privateCA: true`](../installation-references/helm-chart-options.md#常用选项) 是非常重要的。 ::: + 1. 使用 `values.yaml` 文件和当前 Chart 版本升级 Helm 应用程序实例。版本必须匹配以防止 Rancher 升级。 + ```bash helm upgrade rancher rancher-stable/rancher \ --namespace cattle-system \ @@ -164,6 +177,7 @@ kubectl edit -n cattle-system deployment/cattle-cluster-agent 对每个下游集群重复以下步骤： 1. 检索 agent 注册 kubectl 命令： + 1. 找到下游集群的集群 ID (c-xxxxx)。你可以在 Rancher UI 的**集群管理**中查看集群时在浏览器 URL 中找到 ID。 1. 将 Rancher Server URL 和集群 ID 添加到以下 URL：`https:///v3/clusterregistrationtokens?clusterId=`。 1. 复制 `insecureCommand` 字段中的命令，使用此命令是因为未使用私有 CA。 @@ -230,19 +244,24 @@ kubectl -n cattle-system delete secret tls-ca 以下步骤更新了 Rancher Chart 的 Helm 值，因此 Rancher pod 和 Ingress 会使用在步骤 1 中创建的新证书。 1. 调整初始安装期间使用的值，将当前值存储为： + ```bash helm get values rancher -n cattle-system -o yaml > values.yaml ``` + 1. 获取当前部署的 Rancher Chart 的版本字符串： + ```bash helm ls -n cattle-system ``` + 1. 更新 `values.yaml` 文件中的当前 Helm 值： 1. 由于不再使用私有 CA，删除 `privateCA: true` 字段，或将其设置为 `false`。 1. 根据需要调整 `ingress.tls.source` 字段。有关更多信息，请参阅 [Chart 选项](../installation-references/helm-chart-options.md#常用选项)。以下是一些示例： 1. 如果使用公共 CA，继续使用 `secret` 1. 如果使用 Let's Encrypt，将值更新为 `letsEncrypt` 1. 使用 `values.yaml` 文件更新 Rancher Chart 的 Helm 值，并使用当前 Chart 版本防止升级： + ```bash helm upgrade rancher rancher-stable/rancher \ --namespace cattle-system \ diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/upgrade-cert-manager.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/upgrade-cert-manager.md index c2d8a5a59b9..ce1ac727ad6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/upgrade-cert-manager.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/resources/upgrade-cert-manager.md @@ -2,6 +2,12 @@ title: 升级 Cert-Manager --- + + + + +Rancher 适配 API 版本 `cert-manager.io/v1` 并且在 cert-manager v1.13.1 版本上进行了测试。 + Rancher 使用 cert-manager 为 Rancher 高可用部署自动生成和续期 TLS 证书。从 2019 秋季开始，cert-manager 发生了以下的三个重要变更。如果你在此时间段前创建了 Rancher 高可用部署，请进行相关操作。 1. [从 2019 年 11 月 1 日开始，Let's Encrypt 已阻止低于 0.8.0 的 cert-manager 实例。](https://community.letsencrypt.org/t/blocking-old-cert-manager-versions/98753) @@ -92,8 +98,7 @@ Rancher 使用 cert-manager 为 Rancher 高可用部署自动生成和续期 TLS ```plain helm install \ cert-manager jetstack/cert-manager \ - --namespace cert-manager \ - --version v1.11.0 + --namespace cert-manager ``` 1. [恢复备份资源](https://cert-manager.io/docs/tutorials/backup/#restoring-resources)： @@ -125,7 +130,7 @@ Rancher 使用 cert-manager 为 Rancher 高可用部署自动生成和续期 TLS 1. 从 [Helm Chart 仓库](https://artifacthub.io/packages/helm/cert-manager/cert-manager)中获取最新可用的 cert-manager Chart： ```plain - helm fetch jetstack/cert-manager --version v1.11.0 + helm fetch jetstack/cert-manager ``` 1. 使用安装 Chart 的选项来渲染 cert-manager 模板。记住要设置 `image.repository` 选项，以从你的私有镜像仓库拉取镜像。此操作会创建一个包含 Kubernetes manifest 文件的 `cert-manager` 目录。 @@ -140,6 +145,8 @@ Rancher 使用 cert-manager 为 Rancher 高可用部署自动生成和续期 TLS --set cainjector.image.repository=/quay.io/jetstack/cert-manager-cainjector ``` + + Helm 2 命令如下： ```plain @@ -232,11 +239,12 @@ Rancher 使用 cert-manager 为 Rancher 高可用部署自动生成和续期 TLS ``` cmctl upgrade migrate-api-version ``` + 有关详细信息，请参阅 [API 版本迁移文档](https://cert-manager.io/docs/usage/cmctl/#migrate-api-version)。另请参阅[将 1.5 升级到 1.6](https://cert-manager.io/docs/installation/upgrading/upgrading-1.5-1.6/) 和[将 1.6 升级到到 1.7](https://cert-manager.io/docs/installation/upgrading/upgrading-1.6-1.7/)。 1. 正常使用 `helm upgrade` 将 cert-manager 升级到 1.7.1。如果需要，你可以直接从版本 1.5 转到 1.7。 -1. 按照 Helm 教程[更新发布清单的 API 版本](https://helm.sh/docs/topics/kubernetes_apis/#updating-api-versions-of-a-release-manifest)。Chart 发布名称为 `release_name=rancher`，发布命名空​​间为 `release_namespace=cattle-system`。 +1. 按照 Helm 教程[更新发布清单的 API 版本](https://helm.sh/docs/topics/kubernetes_apis/#updating-api-versions-of-a-release-manifest)。Chart 发布名称为 `release_name=rancher`，发布命名空 ​​ 间为 `release_namespace=cattle-system`。 1. 在解码后的文件中，搜索 `cert-manager.io/v1beta1` 并将其**替换**为 `cert-manager.io/v1`。 @@ -278,4 +286,3 @@ v0.11 版本标志着删除先前 Cert-Manager 版本中使用的 v1alpha1 API 如需了解变更细节以及迁移说明，请参见[将 Cert-Manager 从 v0.10 升级到 v0.11](https://cert-manager.io/docs/installation/upgrading/upgrading-0.10-0.11/)。 如需获得更多信息，请参见 [Cert-Manager 升级](https://cert-manager.io/docs/installation/upgrade/)。 - diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md index aa851f3a50c..9ce08754a1f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/upgrade-and-roll-back-kubernetes.md @@ -2,11 +2,14 @@ title: 升级和回滚 Kubernetes --- + + + + 升级到最新版本的 Rancher 之后，下游 Kubernetes 集群可以升级为 Rancher 支持的最新的 Kubernetes 版本。 Rancher 使用 RKE（Rancher Kubernetes Engine）来预置和编辑 RKE 集群。有关为 RKE 集群配置升级策略的更多信息，请参阅 [RKE 文档](https://rancher.com/docs/rke/latest/en/)。 - ## 经过测试的 Kubernetes 版本 Rancher 在发布新版本之前，会对其与 Kubernetes 的最新次要版本进行测试，以确保兼容性。有关各个 Rancher 版本测试了哪些 Kubernetes 版本的详细信息，请参阅[支持维护条款](https://rancher.com/support-maintenance-terms/all-supported-versions/rancher-v2.6.0/)。 @@ -17,7 +20,6 @@ RKE v1.1.0 改变了集群升级的方式。 在 [RKE 文档](https://rancher.com/docs/rke/latest/en/upgrades/how-upgrades-work)中，你将了解编辑或升级 RKE Kubernetes 集群时会发生的情况。 - ## 升级的最佳实践 在升级集群的 Kubernetes 版本时，我们建议你： @@ -32,9 +34,9 @@ RKE v1.1.0 改变了集群升级的方式。 :::note 先决条件： -- 以下选项适用于 [Rancher 启动的 Kubernetes 集群](../../pages-for-subheaders/launch-kubernetes-with-rancher.md)和[注册的 K3s Kubernetes 集群](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/register-existing-clusters.md#已注册-rke2-和-k3s-集群的附加功能)。 +- 以下选项适用于 [Rancher 启动的 Kubernetes 集群](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)和[注册的 K3s Kubernetes 集群](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/register-existing-clusters.md#已注册-rke2-和-k3s-集群的附加功能)。 - 以下选项也适用于导入且已注册的 RKE2 集群。如果你从外部云平台导入集群但不注册，你将无法在 Rancher 中升级 Kubernetes 版本。 -- 在升级 Kubernetes 之前，先[备份你的集群](../../pages-for-subheaders/backup-restore-and-disaster-recovery.md)。 +- 在升级 Kubernetes 之前，先[备份你的集群](../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md)。 ::: @@ -75,7 +77,7 @@ RKE v1.1.0 改变了集群升级的方式。 ### 使用 Rancher UI 在升级期间启用节点清空 -默认情况下，RKE 会在升级之前[封锁](https://kubernetes.io/docs/concepts/architecture/nodes/#manual-node-administration)每个节点。默认情况下，[清空](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/)会在升级期间被禁用。如果在集群配置中启用了清空​​，RKE 将在升级之前对节点进行封锁和清空。 +默认情况下，RKE 会在升级之前[封锁](https://kubernetes.io/docs/concepts/architecture/nodes/#manual-node-administration)每个节点。默认情况下，[清空](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/)会在升级期间被禁用。如果在集群配置中启用了清空 ​​，RKE 将在升级之前对节点进行封锁和清空。 要在集群升级期间清空每个节点： diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md index d8021a77f4f..30c5baffa62 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md @@ -2,6 +2,10 @@ title: 在不升级 Rancher 的情况下升级 Kubernetes --- + + + + RKE 元数据功能允许你在新版本 Kubernetes 发布后立即为集群配置新版本，而无需升级 Rancher。此功能对于使用 Kubernetes 的补丁版本非常有用，例如，在原本支持 Kubernetes v1.14.6 的 Rancher Server 版本中，将 Kubernetes 升级到 v1.14.7。 :::note @@ -10,7 +14,7 @@ Kubernetes API 可以在次要版本之间更改。因此，我们不支持引 ::: -Rancher 的 Kubernetes 元数据包含 Rancher 用于配置 [RKE 集群](../../pages-for-subheaders/launch-kubernetes-with-rancher.md)的 Kubernetes 版本信息。Rancher 会定期同步数据并为 **系统镜像**、**服务选项**和**插件模板**创建自定义资源定义 (CRD)。因此，当新的 Kubernetes 版本与 Rancher Server 版本兼容时，Kubernetes 元数据可以使 Rancher 使用新版本来配置集群。元数据概述了 [Rancher Kubernetes Engine](https://rancher.com/docs/rke/latest/en/) (RKE) 用于部署各种 Kubernetes 版本的信息。 +Rancher 的 Kubernetes 元数据包含 Rancher 用于配置 [RKE 集群](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)的 Kubernetes 版本信息。Rancher 会定期同步数据并为 **系统镜像**、**服务选项**和**插件模板**创建自定义资源定义 (CRD)。因此，当新的 Kubernetes 版本与 Rancher Server 版本兼容时，Kubernetes 元数据可以使 Rancher 使用新版本来配置集群。元数据概述了 [Rancher Kubernetes Engine](https://rancher.com/docs/rke/latest/en/) (RKE) 用于部署各种 Kubernetes 版本的信息。 下表描述了受周期性数据同步影响的 CRD。 @@ -20,11 +24,11 @@ Rancher 的 Kubernetes 元数据包含 Rancher 用于配置 [RKE 集群](../../p ::: -| 资源 | 描述 | Rancher API URL | -|----------|-------------|-----------------| -| 系统镜像 | 用于通过 RKE 部署 Kubernetes 集群的系统镜像列表。 | `/v3/rkek8ssystemimages` | -| 服务选项 | 传递给 Kubernetes 组件的默认选项，例如 `kube-api`、`scheduler`、`kubelet`、`kube-proxy` 和 `kube-controller-manager` | `/v3/rkek8sserviceoptions` | -| 插件模板 | 用于部署插件组件的 YAML 定义，例如 Canal、Calico、Flannel、Weave、Kube-dns、CoreDNS、`metrics-server`、`nginx-ingress` | `/v3/rkeaddons` | +| 资源 | 描述 | Rancher API URL | +| -------- | ---------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | +| 系统镜像 | 用于通过 RKE 部署 Kubernetes 集群的系统镜像列表。 | `/v3/rkek8ssystemimages` | +| 服务选项 | 传递给 Kubernetes 组件的默认选项，例如 `kube-api`、`scheduler`、`kubelet`、`kube-proxy` 和 `kube-controller-manager` | `/v3/rkek8sserviceoptions` | +| 插件模板 | 用于部署插件组件的 YAML 定义，例如 Canal、Calico、Flannel、Weave、Kube-dns、CoreDNS、`metrics-server`、`nginx-ingress` | `/v3/rkeaddons` | 管理员可以通过配置 RKE 元数据设置来执行以下操作： @@ -48,7 +52,7 @@ Rancher 的 Kubernetes 元数据包含 Rancher 用于配置 [RKE 集群](../../p :::caution -只有管​​理员可以更改这些设置。 +只有管 ​​ 理员可以更改这些设置。 ::: @@ -64,6 +68,7 @@ RKE 元数据的配置控制 Rancher 同步元数据的频率以及从何处下 - `refresh-interval-minutes`：Rancher 等待同步元数据的时间。如果要禁用定期刷新，请将 `refresh-interval-minutes` 设置为 0。 - `url`：Rancher 从中获取数据的 HTTP 路径。该路径必须是 JSON 文件的直接路径。例如，Rancher v2.4 的默认 URL 是 `https://releases.rancher.com/kontainer-driver-metadata/release-v2.4/data.json`。 + 1. 单击**保存**。 如果你没有离线设置，则无需指定 Rancher 获取元数据的 URL，因为默认是从 [Rancher 的元数据 Git 仓库获取](https://github.com/rancher/kontainer-driver-metadata/blob/dev-v2.5/data/data.json)的。 @@ -80,7 +85,10 @@ Rancher Server 会定期刷新 `rke-metadata-config` 来下载新的 Kubernetes 在将新的 Kubernetes 版本加载到 Rancher Server 中之后，需要执行其他步骤才能使用它们启动集群。Rancher 需要访问更新的系统镜像。虽然只有管理员可以更改元数据设置，但任何用户都可以下载 Rancher 系统镜像并为镜像准备私有容器镜像仓库。 -1. 要把系统镜像下载到私有镜像仓库，请单击 Rancher UI 左下角的 Rancher Server 版本。 +要下载私有镜像仓库的系统镜像： + +1. 点击左上角的 **☰**。 +1. 点击左侧导航底部的**简介**。 1. 下载适用于 Linux 或 Windows 操作系统的镜像。 1. 下载 `rancher-images.txt`。 1. 使用[离线环境安装](other-installation-methods/air-gapped-helm-cli-install/publish-images.md)时使用的步骤准备私有镜像仓库，但不要使用发布页面中的 `rancher-images.txt`，而是使用上一个步骤中获取的文件。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/overview.md index 903c8c28bd2..1f3b3afa943 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/overview.md @@ -2,6 +2,10 @@ title: 概述 --- + + + + Rancher 是一个为使用容器的公司打造的容器管理平台。Rancher 使得开发者可以随处运行 Kubernetes（Run Kubernetes Everywhere），满足 IT 需求规范，赋能 DevOps 团队。 ## Run Kubernetes Everywhere @@ -30,21 +34,21 @@ Rancher API Server 是基于嵌入式 Kubernetes API Server 和 etcd 数据库 ### 授权和基于角色的权限控制（RBAC） -- **用户管理**：Rancher API Server 除了管理本地用户，还[管理用户用来访问外部服务所需的认证信息](../pages-for-subheaders/authentication-config.md)，如登录 Active Directory 和 GitHub 所需的账号密码。 -- **授权**：Rancher API Server 可以管理[访问控制策略](../pages-for-subheaders/manage-role-based-access-control-rbac.md)和[安全策略](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md)。 +- **用户管理**：Rancher API Server 除了管理本地用户，还[管理用户用来访问外部服务所需的认证信息](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md)，如登录 Active Directory 和 GitHub 所需的账号密码。 +- **授权**：Rancher API Server 可以管理[访问控制策略](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md)和[安全策略](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md)。 ### 使用 Kubernetes 的功能 -- **配置 Kubernetes 集群**：Rancher API Server 可以在已有节点上[配置 Kubernetes](../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)，或进行 [Kubernetes 版本升级](installation-and-upgrade/upgrade-and-roll-back-kubernetes.md)。 -- **管理应用商店**：Rancher 支持使用 [Helm Chart 应用商店](../pages-for-subheaders/helm-charts-in-rancher.md)实现轻松重复部署应用。 -- **管理项目**：项目由集群中多个命名空间和访问控制策略组成，是 Rancher 中的一个概念，Kubernetes 中并没有这个概念。你可以使用项目实现以组为单位，管理多个命名空间，并进行 Kubernetes 相关操作。Rancher UI 提供用于[项目管理](../pages-for-subheaders/manage-projects.md)和[项目内应用管理](../pages-for-subheaders/kubernetes-resources-setup.md)的功能。 -- **Fleet 持续交付**：在 Rancher 中，你可以使用 [Fleet 持续交付](../pages-for-subheaders/fleet-gitops-at-scale.md)将应用程序从 Git 仓库部署到目标下游 Kubernetes 集群，无需任何手动操作。 -- **Istio**：[Rancher 与 Istio 集成](../pages-for-subheaders/istio.md)，使得管理员或集群所有者可以将 Istio 交给开发者，然后开发者使用 Istio 执行安全策略，排查问题，或为蓝绿部署，金丝雀部署，和 A/B 测试进行流量管理。 +- **配置 Kubernetes 集群**：Rancher API Server 可以在已有节点上[配置 Kubernetes](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)，或进行 [Kubernetes 版本升级](installation-and-upgrade/upgrade-and-roll-back-kubernetes.md)。 +- **管理应用商店**：Rancher 支持使用 [Helm Chart 应用商店](../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md)实现轻松重复部署应用。 +- **管理项目**：项目由集群中多个命名空间和访问控制策略组成，是 Rancher 中的一个概念，Kubernetes 中并没有这个概念。你可以使用项目实现以组为单位，管理多个命名空间，并进行 Kubernetes 相关操作。Rancher UI 提供用于[项目管理](../how-to-guides/advanced-user-guides/manage-projects/manage-projects.md)和[项目内应用管理](../how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md)的功能。 +- **Fleet 持续交付**：在 Rancher 中，你可以使用 [Fleet 持续交付](../integrations-in-rancher/fleet/fleet.md)将应用程序从 Git 仓库部署到目标下游 Kubernetes 集群，无需任何手动操作。 +- **Istio**：[Rancher 与 Istio 集成](../integrations-in-rancher/istio/istio.md)，使得管理员或集群所有者可以将 Istio 交给开发者，然后开发者使用 Istio 执行安全策略，排查问题，或为蓝绿部署，金丝雀部署，和 A/B 测试进行流量管理。 ### 配置云基础设施 - **同步节点信息**：Rancher API Server 可以同步所有集群中全部[节点](../how-to-guides/new-user-guides/manage-clusters/nodes-and-node-pools.md)的信息。 -- **配置云基础设施**：如果你为 Rancher 配置了云提供商，Rancher 可以在云端动态配置[新节点](../pages-for-subheaders/use-new-nodes-in-an-infra-provider.md)和[持久化存储](../pages-for-subheaders/create-kubernetes-persistent-storage.md)。 +- **配置云基础设施**：如果你为 Rancher 配置了云提供商，Rancher 可以在云端动态配置[新节点](../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)和[持久化存储](../how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md)。 ### 查看集群信息 @@ -54,9 +58,9 @@ Rancher API Server 是基于嵌入式 Kubernetes API Server 和 etcd 数据库 ## 使用 Rancher 编辑下游集群 -对于已有集群而言，可提供的选项和设置取决于你配置集群的方法。例如，只有[通过 RKE 启动](../pages-for-subheaders/launch-kubernetes-with-rancher.md)的集群才有可编辑的**集群选项**。 +对于已有集群而言，可提供的选项和设置取决于你配置集群的方法。例如，只有[通过 RKE 启动](../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)的集群才有可编辑的**集群选项**。 -使用 Rancher 创建集群后，集群管理员可以管理集群成员，管理节点池，或进行[其他操作](../pages-for-subheaders/cluster-configuration.md)。 +使用 Rancher 创建集群后，集群管理员可以管理集群成员，管理节点池，或进行[其他操作](../reference-guides/cluster-configuration/cluster-configuration.md)。 下表总结了每一种类型的集群和对应的可编辑的选项和设置： diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/aws-marketplace.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/aws-marketplace.md index 2fc1ce0bd33..1d10accb91d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/aws-marketplace.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/aws-marketplace.md @@ -1,6 +1,10 @@ --- -title: Rancher AWS Marketplace 快速入门 -description: 使用 Amazon EKS 部署 Rancher Server。 +title: Rancher Prime AWS Marketplace 快速入门 +description: 使用 Amazon Marketplace 列表部署 Rancher Server。 --- -Amazon Elastic Kubernetes Service (EKS) 可以快速[将 Rancher 部署到 Amazon Web Services (AWS)](https://documentation.suse.com/trd/kubernetes/single-html/gs_rancher_aws-marketplace/)。详情请参见我们的 [Amazon Marketplace 列表](https://aws.amazon.com/marketplace/pp/prodview-go7ent7goo5ae)。观看 [demo](https://youtu.be/9dznJ7Ons0M)，了解 AWS Marketplace SUSE Rancher 设置的演练。 + + + + +你可以快速将 Rancher Prime 部署到 Amazon Elastic Kubernetes Service (EKS)。详情请参见[说明](https://suse-enceladus.github.io/marketplace-docs/rancher-prime/aws/?repository=rancher-payg-billing-adapter-llc-prd)以及我们的 [Amazon Marketplace 列表](https://aws.amazon.com/marketplace/pp/prodview-go7ent7goo5ae)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/aws.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/aws.md index cb47333ce32..5e9a3d4d22e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/aws.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/aws.md @@ -3,11 +3,15 @@ title: Rancher AWS 快速入门指南 description: 阅读此分步 Rancher AWS 指南，以快速部署带有单节点下游 Kubernetes 集群的 Rancher Server。 --- + + + + 你可以参考以下步骤，在 AWS 的单节点 K3s Kubernetes 集群中快速部署 Rancher Server，并附加一个单节点下游 Kubernetes 集群。 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -86,7 +90,7 @@ AWS 模块只创建一个 EC2 密钥对、一个 EC2 安全组和一个 EC2 实 ## 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/azure.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/azure.md index 33a8d411ba0..5ff3cdee98d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/azure.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/azure.md @@ -3,11 +3,15 @@ title: Rancher Azure 快速入门指南 description: 阅读此分步 Rancher Azure 指南，以快速部署带有单节点下游 Kubernetes 集群的 Rancher Server。 --- + + + + 你可以参考以下步骤，在 Azure 的单节点 K3s Kubernetes 集群中快速部署 Rancher Server，并附加一个单节点下游 Kubernetes 集群。 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -72,7 +76,7 @@ description: 阅读此分步 Rancher Azure 指南，以快速部署带有单节 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager.md new file mode 100644 index 00000000000..d744408670c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/deploy-rancher-manager.md @@ -0,0 +1,24 @@ +--- +title: 部署 Rancher Server +--- + + + + + +你可使用以下指南之一，在你选择的提供商中部署和配置 Rancher 和 Kubernetes 集群。 + +- [AWS](aws.md)（使用 Terraform） +- [AWS Marketplace](aws-marketplace.md)（使用 Amazon EKS） +- [Azure](azure.md)（使用 Terraform） +- [DigitalOcean](digitalocean.md)（使用 Terraform） +- [GCP](gcp.md)（使用 Terraform） +- [Hetzner Cloud](hetzner-cloud.md)（使用 Terraform） +- [Linode](linode.md) (使用 Terraform) +- [Vagrant](vagrant.md) +- [Equinix Metal](equinix-metal.md) +- [Outscale](outscale-qs.md)（使用 Terraform） + +如有需要，你可以查看以下指南以了解分步步骤。如果你需要在其他提供商中或本地运行 Rancher，或者你只是想看看它是多么容易上手，你可阅读以下指南： + +- [手动安装](helm-cli.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md index e87f17c26d0..c93813c1204 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/digitalocean.md @@ -3,11 +3,15 @@ title: Rancher DigitalOcean 快速入门指南 description: 阅读此分步 Rancher DigitalOcean 指南，以快速部署带有单节点下游 Kubernetes 集群的 Rancher Server。 --- + + + + 你可以参考以下步骤，在 DigitalOcean 的单节点 K3s Kubernetes 集群中快速部署 Rancher Server，并附加一个单节点下游 Kubernetes 集群。 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -65,7 +69,7 @@ description: 阅读此分步 Rancher DigitalOcean 指南，以快速部署带有 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/equinix-metal.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/equinix-metal.md index b22ff8d5ff3..2a6562d2d34 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/equinix-metal.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/equinix-metal.md @@ -2,6 +2,10 @@ title: Rancher Equinix Metal 快速入门 --- + + + + ## 本章节引导你： - 配置 Equinix Metal server @@ -11,7 +15,7 @@ title: Rancher Equinix Metal 快速入门 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。不建议将 Docker 安装用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。不建议将 Docker 安装用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -29,7 +33,7 @@ title: Rancher Equinix Metal 快速入门 ### 1. 配置 Equinix Metal 主机 -开始部署 Equinix Metal 主机。你可以使用 Equinix Metal 控制台，API 或 CLI 来配置 Equinix Metal Server。如果你需要了解每种 Deployment 类型的说明，请参见 [Equinix Metal 部署](https://metal.equinix.com/developers/docs/deploy/on-demand/)。以下链接介绍 Equinix Metal Server 的类型以及价格： +开始部署 Equinix Metal 主机。你可以使用 Equinix Metal 控制台、CLI 或 API 来配置 Equinix Metal Server。如果你需要了解每种 Deployment 类型的说明，请参见 [Equinix Metal 部署](https://metal.equinix.com/developers/docs/deploy/on-demand/)。以下链接介绍 Equinix Metal Server 的类型以及价格： - [Equinix Metal Server 类型](https://metal.equinix.com/developers/docs/servers/about/) - [Equinix Metal 价格](https://metal.equinix.com/developers/docs/servers/server-specs/) @@ -38,7 +42,7 @@ title: Rancher Equinix Metal 快速入门 - 如果使用 CLI 或 API 配置新的 Equinix Metal Server，你需要提供项目 ID、计划、metro 和操作系统。 - 当使用云主机的虚拟机时，你需要允许 80 和 443 端口的入站 TCP 通信。有关端口配置的信息，请参见你的云主机的文档。 - 如需了解所有端口要求，请参见 [Docker 安装](../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)。 -- 根据我们的[要求](../../../pages-for-subheaders/installation-requirements.md)配置主机。 +- 根据我们的[要求](../../installation-and-upgrade/installation-requirements/installation-requirements.md)配置主机。 ::: ### 2. 安装 Rancher @@ -103,4 +107,4 @@ title: Rancher Equinix Metal 快速入门 #### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md index 5c113fc447c..cdbfebe93dd 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/gcp.md @@ -3,11 +3,15 @@ title: Rancher GCP 快速入门指南 description: 阅读此分步 Rancher GCP 指南，以快速部署带有单节点下游 Kubernetes 集群的 Rancher Server。 --- + + + + 你可以参考以下步骤，在 GCP 的单节点 K3s Kubernetes 集群中快速部署 Rancher Server，并附加一个单节点下游 Kubernetes 集群。 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -68,7 +72,7 @@ description: 阅读此分步 Rancher GCP 指南，以快速部署带有单节点 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md index b3acb4f9ecc..221b27292df 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli.md @@ -2,6 +2,10 @@ title: Helm CLI 快速入门 --- + + + + 本文提供了快速安装 Rancher 的方法。 这些说明假设你有一个 Linux 虚拟机，并能从本地工作站与之通信。Rancher 将安装在 Linux 主机上。你将需要检索该主机的 IP 地址，以便从本地工作站访问 Rancher。Rancher 旨在远程管理 Kubernetes 集群，因此 Rancher 管理的任何 Kubernetes 集群也都需要能够访问该 IP 地址。 @@ -10,7 +14,7 @@ title: Helm CLI 快速入门 你的 Linux 主机可以位于任何地方。例如，它可以是 Amazon EC2 实例、Digital Ocean Droplet 或 Azure 虚拟机。其他 Rancher 文档也经常称它们为“节点”。部署 Linux 主机的一种方法是设置一个 Amazon EC2 实例，如[本教程](../../../how-to-guides/new-user-guides/infrastructure-setup/nodes-in-amazon-ec2.md)中所示。 -完整的安装要求在[这里](../../../pages-for-subheaders/installation-requirements.md)。 +完整的安装要求在[这里](../../installation-and-upgrade/installation-requirements/installation-requirements.md)。 ## 在 Linux 上安装 K3s @@ -104,7 +108,7 @@ helm repo add rancher-latest https://releases.rancher.com/server-charts/latest kubectl create namespace cattle-system -kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.11.0/cert-manager.crds.yaml +kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download//cert-manager.crds.yaml helm repo add jetstack https://charts.jetstack.io @@ -112,14 +116,12 @@ helm repo update helm install cert-manager jetstack/cert-manager \ --namespace cert-manager \ - --create-namespace \ - --version v1.11.0 + --create-namespace # Windows Powershell helm install cert-manager jetstack/cert-manager ` --namespace cert-manager ` - --create-namespace ` - --version v1.11.0 + --create-namespace ``` 安装 Rancher 的最终命令如下。该命令需要一个将流量转发到 Linux 主机的域名。为了简化本教程，你可以使用假域名。`.sslip.io` 是一个假域名的例子。 @@ -149,6 +151,6 @@ helm install rancher rancher-latest/rancher ` 为了简化说明，我们使用了一个假域名和自签名证书来进行安装。因此，你可能需要在 Web 浏览器中添加一个安全例外来查看 Rancher UI。请注意，对于生产安装，你需要具有负载均衡器、真实域名和真实证书的高可用性设置。 -这些说明还省略了完整的安装要求和其他安装选项。如果你对这些步骤有任何疑问，请参阅完整的 [Helm CLI 安装文档](../../../pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster.md)。 +这些说明还省略了完整的安装要求和其他安装选项。如果你对这些步骤有任何疑问，请参阅完整的 [Helm CLI 安装文档](../../installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/install-upgrade-on-a-kubernetes-cluster.md)。 -要使用新的 Rancher Server 来启动新的 Kubernetes 集群，你可能需要在 Rancher 中设置云凭证。有关更多信息，请参阅[使用 Rancher 启动 Kubernetes 集群](../../../pages-for-subheaders/launch-kubernetes-with-rancher.md)。 +要使用新的 Rancher Server 来启动新的 Kubernetes 集群，你可能需要在 Rancher 中设置云凭证。有关更多信息，请参阅[使用 Rancher 启动 Kubernetes 集群](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md index 585353441b2..6f1421a54bc 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/hetzner-cloud.md @@ -3,11 +3,15 @@ title: Rancher Hetzner Cloud 快速入门指南 description: 阅读此分步 Rancher Hetzner Cloud 指南，以快速部署带有单节点下游 Kubernetes 集群的 Rancher Server。 --- + + + + 你可以参考以下步骤，在 Hetzner Cloud 的单节点 K3s Kubernetes 集群中快速部署 Rancher Server，并附加一个单节点下游 Kubernetes 集群。 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -67,7 +71,7 @@ description: 阅读此分步 Rancher Hetzner Cloud 指南，以快速部署带 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/linode.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/linode.md new file mode 100644 index 00000000000..54b0b3a25f1 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/linode.md @@ -0,0 +1,83 @@ +--- +title: Rancher Linode 快速入门指南 +description: 阅读此分步 Rancher Linode 指南，以快速部署带有单节点下游 Kubernetes 集群的 Rancher Server。 +--- + + + + + +你可以参考以下步骤，在 Linode 的单节点 K3s Kubernetes 集群中快速部署 Rancher Server，并附加一个单节点下游 Kubernetes 集群。 + +:::caution + +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 + +::: + +## 先决条件 + +:::caution + +部署到 Linode 会产生费用。 + +::: + +- [Linode 账号](https://linode.com): 用于运行服务器和集群。 +- [Linode 访问密钥](https://www.linode.com/docs/products/tools/api/guides/manage-api-tokens/): 用于权限认证的 Linode 访问密钥。 +- [Terraform](https://www.terraform.io/downloads.html): 用于在 Linode 中配置服务器和集群。 + + +## 开始使用 + +1. 使用命令行工具，执行 `git clone https://github.com/rancher/quickstart` 把 [Rancher Quickstart](https://github.com/rancher/quickstart) 克隆到本地。 + +2. 执行 `cd quickstart/rancher/linode` 命令，进入包含 Terraform 文件的 Linode 文件夹。 + +3. 把 `terraform.tfvars.example` 文件重命名为 `terraform.tfvars`。 + +4. 编辑 `terraform.tfvars` 文件，并替换以下变量： + - `linode_token` - 上面提到的 Linode 访问密钥。 + - `rancher_server_admin_password` - 替换为创建 Rancher Server 的 admin 账号的密码（最少 12 字符） + +5. **可选**：修改 `terraform.tfvars` 中的可选参数。 + 参见 [Quickstart Readme](https://github.com/rancher/quickstart) 以及 [Linode Quickstart Readme](https://github.com/rancher/quickstart/tree/master/rancher/linode) 了解更多信息。 + 建议包括： + - `linode_region` - 创建服务器以及集群的目标 Linode 区域。 + - 默认: `eu-central` + - 完整的区域列表, 请参照[官方的可用区域页面](https://www.linode.com/global-infrastructure/availability/). + - `prefix` - 所有创建资源的前缀 + - `linode_type` - 所有的 Linode 资源使用的类型/计划 + - 默认: `g6-standard-2` + - 完整的计划列表, 请参照[官方的计划类型页面](https://www.linode.com/docs/products/compute/compute-instances/plans/). + +6. 执行 `terraform init`。 + +7. 执行 `terraform apply --auto-approve` 以初始化环境。然后，等待命令行工具返回以下信息： + + ``` + Apply complete! Resources: 15 added, 0 changed, 0 destroyed. + + Outputs: + + rancher_node_ip = xx.xx.xx.xx + rancher_server_url = https://rancher.xx.xx.xx.xx.sslip.io + workload_node_ip = yy.yy.yy.yy + ``` + +8. 将以上输出中的 `rancher_server_url` 粘贴到浏览器中。在登录页面中登录（默认用户名为 `admin`，密码为在 `rancher_server_admin_password` 中设置的密码）。 +9. 使用 `quickstart/rancher/linode` 中生成的 `id_rsa` 密钥 SSH 到 Rancher Server。 + +#### 结果 + +两个 Kubernetes 集群已部署到你的 Linode 账户中，一个运行 Rancher Server，另一个为实验部署做好准备。请注意，虽然这种设置是探索 Rancher 功能的好方法，但在生产环境中，应遵循我们的高可用设置指南。用于虚拟机的 SSH 密钥是自动生成的，存储在模块目录中。 + +### 后续操作 + +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 + +## 销毁环境 + +1. 进入 `quickstart/rancher/linode` 文件夹，然后执行 `terraform destroy --auto-approve`。 + +2. 等待命令行界面显示资源已删除的消息。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md index 46b42901db9..2906b5fcea8 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/outscale-qs.md @@ -3,11 +3,15 @@ title: Rancher Outscale 快速入门指南 description: 阅读此分步 Rancher Outscale 指南，以快速部署带有单节点下游 Kubernetes 集群的 Rancher Server。 --- + + + + 你可以参考以下步骤，在 Outscale 的单节点 K3s Kubernetes 集群中快速部署 Rancher Server，并附加一个单节点下游 Kubernetes 集群。 :::note -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -67,7 +71,7 @@ description: 阅读此分步 Rancher Outscale 指南，以快速部署带有单 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/vagrant.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/vagrant.md index dcddb74cfe9..5f60d818913 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/vagrant.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-rancher-manager/vagrant.md @@ -2,11 +2,15 @@ title: Vagrant 快速入门 --- + + + + 你可以参考以下步骤快速部署 Rancher Server，并附加一个单节点集群。 :::caution -本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../../pages-for-subheaders/installation-and-upgrade.md)。 +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../../installation-and-upgrade/installation-and-upgrade.md)。 ::: @@ -17,11 +21,12 @@ title: Vagrant 快速入门 - 至少 4GB 的可用内存。 ### 注意 + - Vagrant 需要使用插件来创建 VirtualBox 虚拟机。请执行以下命令进行安装： - `vagrant plugin install vagrant-vboxmanage` + `vagrant plugin install vagrant-vboxmanage` - `vagrant plugin install vagrant-vbguest` + `vagrant plugin install vagrant-vbguest` ## 开始使用 @@ -42,7 +47,7 @@ title: Vagrant 快速入门 ### 后续操作 -使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../../../pages-for-subheaders/deploy-rancher-workloads.md)。 +使用 Rancher 创建 deployment。详情请参见[创建 Deployment](../deploy-workloads/deploy-workloads.md)。 ## 销毁环境 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md new file mode 100644 index 00000000000..b0aa55e4fb4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/deploy-workloads/deploy-workloads.md @@ -0,0 +1,12 @@ +--- +title: 部署工作负载 +--- + + + + + +这些指南指导你完成一个应用的部署，包括如何将应用暴露在集群之外使用。 + +- [部署带有 Ingress 的工作负载](workload-ingress.md) +- [部署带有 NodePort 的工作负载](nodeports.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/quick-start-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/quick-start-guides.md new file mode 100644 index 00000000000..b33df9c46bc --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/getting-started/quick-start-guides/quick-start-guides.md @@ -0,0 +1,21 @@ +--- +title: Rancher 部署快速入门指南 +--- + + + + + +:::caution + +本章节中提供的指南，旨在帮助你快速启动一个用于 Rancher 的沙盒，以评估 Rancher 是否能满足你的使用需求。快速入门指南不适用于生产环境。如果你需要获取生产环境的操作指导，请参见[安装](../installation-and-upgrade/installation-and-upgrade.md)。 + +::: + +你可以阅读本章节，以快速开始部署和测试 Rancher 2.x。本章节包含 Rancher 的简单设置和一些常见用例的说明。未来，我们会在本章节中添加更多内容。 + +我们提供以下快速入门指南： + +- [部署 Rancher Server](deploy-rancher-manager/deploy-rancher-manager.md)：使用最方便的方式运行 Rancher。 + +- [部署工作负载](deploy-workloads/deploy-workloads.md)：部署一个简单的[工作负载](https://kubernetes.io/docs/concepts/workloads/)并公暴露工作负载，以从集群外部访问工作负载。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/advanced-user-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/advanced-user-guides.md new file mode 100644 index 00000000000..80d6b10f321 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/advanced-user-guides.md @@ -0,0 +1,11 @@ +--- +title: 高级用户指南 +--- + + + + + +高级用户指南是“问题导向”的文档，用户可以从中学习如何解决问题。高级用于指南与新用户指南的主要区别在于，高级用户指南面向更有经验或更高级的用户，这些用户对文档有更多的技术需求，而且已经了解 Rancher 及其功能。他们知道自己需要做什么，只是需要额外的指导来完成更复杂的任务。 + +应该注意的是，新用户指南和高级用户指南都没有提供详细的解释或讨论（这些文档不包括在本部分）。操作指南侧重于引导用户通过可重复、有效的步骤来学习新技能、掌握某些操作或解决某些问题。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md new file mode 100644 index 00000000000..ff6b6200e19 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md @@ -0,0 +1,17 @@ +--- +title: CIS 扫描指南 +--- + + + + + +- [安装 Rancher CIS Benchmark](install-rancher-cis-benchmark.md) +- [卸载 Rancher CIS Benchmark](uninstall-rancher-cis-benchmark.md) +- [运行扫描](run-a-scan.md) +- [定时运行扫描](run-a-scan-periodically-on-a-schedule.md) +- [跳过测试](skip-tests.md) +- [查看报告](view-reports.md) +- [为 Rancher CIS Benchmark 启用告警](enable-alerting-for-rancher-cis-benchmark.md) +- [为定时扫描配置告警](configure-alerts-for-periodic-scan-on-a-schedule.md) +- [为集群扫描创建自定义 Benchmark 版本](create-a-custom-benchmark-version-to-run.md) \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters.md new file mode 100644 index 00000000000..12925c800a9 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/enable-api-audit-log-in-downstream-clusters.md @@ -0,0 +1,140 @@ +--- +title: 下游集群开启 API 审计日志 +--- + + + + + +Kubernetes 审计提供了由 Kube-apiserver 执行的与安全相关的、按时间顺序排列的集群审计记录。Kube API 会在请求执行的每个阶段都生成一个事件，然后根据策略进行预处理并保存，审计策略配置了要记录的内容。 + +你可能希望将审计日志配置为遵守互联网安全中心 (CIS) Kubernetes 基准控制的一部分。 + +有关配置的详细信息，请参阅 [Kubernetes 官方文档](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/)。 + + + + +:::note + +Rancher v2.7.2 及以上版本提供此功能。 + +::: + +首先，你需要创建一个 Secret 或 ConfigMap，用于配置审计策略。 + +Secret 或 ConfigMap 必须满足以下两个要求： + +1. 必须位于 Cluster 对象所在的 `fleet-default` 命名空间中。 +2. 它必须具有注释 `rke.cattle.io/object-authorized-for-clusters: cluster-name1,cluster-name2`，以允许目标集群使用它。 + +:::tip + +Rancher Dashboard 提供了易于使用的表单页面用于创建 Secret 或 ConfigMap。 + +::: + +例子： + +```yaml +apiVersion: v1 +data: + audit-policy: >- + IyBMb2cgYWxsIHJlcXVlc3RzIGF0IHRoZSBNZXRhZGF0YSBsZXZlbC4KYXBpVmVyc2lvbjogYXVkaXQuazhzLmlvL3YxCmtpbmQ6IFBvbGljeQpydWxlczoKLSBsZXZlbDogTWV0YWRhdGE= +kind: Secret +metadata: + annotations: + rke.cattle.io/object-authorized-for-clusters: cluster1 + name: name1 + namespace: fleet-default +``` + +可以通过编辑集群 YAML 的 `machineSelectorFiles` 和 `machineGlobalConfig` 字段来启用和配置审计日志。 + +例子： + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +spec: + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - audit-policy-file=/dev-audit-policy.yaml + - audit-log-path=/dev-audit.logs + machineSelectorFiles: + - fileSources: + - configMap: + name: '' + secret: + items: + - key: audit-policy + path: /dev-audit-policy.yaml + name: dev-audit-policy + machineLabelSelector: + matchLabels: + rke.cattle.io/control-plane-role: 'true' +``` + +有关集群配置的更多信息，请参阅 REK2 或 K3s 集群配置参考页。 + + + + + +可通过编辑集群 YAML 来启用和配置审计日志。 + +在启用审计日志后，将使用 RKE1 的默认值。 + +```yaml +# +# Rancher Config +# +rancher_kubernetes_engine_config: + services: + kube-api: + audit_log: + enabled: true +``` + +你还可以自定义审计日志配置。 + +```yaml +# +# Rancher Config +# +rancher_kubernetes_engine_config: + services: + kube-api: + audit_log: + enabled: true + configuration: + max_age: 6 + max_backup: 6 + max_size: 110 + path: /var/log/kube-audit/audit-log.json + format: json + policy: + apiVersion: audit.k8s.io/v1 # 这里必须填写 + kind: Policy + omitStages: + - "RequestReceived" + rules: + # Log pod changes at RequestResponse level + - level: RequestResponse + resources: + - group: "" + # Resource "pods" doesn't match requests to any subresource of pods, + # which is consistent with the RBAC policy. + resources: ["pods"] + # Log "pods/log", "pods/status" at Metadata level + - level: Metadata + resources: + - group: "" + resources: ["pods/log", "pods/status"] +``` + +配置详情请参考 [RKE1 官方文档](https://rke.docs.rancher.com/config-options/audit-log)。 + + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md new file mode 100644 index 00000000000..710abfc58ec --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md @@ -0,0 +1,127 @@ +--- +title: 启用实验功能 +--- + + + + + +Rancher 包含一些默认关闭的实验功能。在某些情况下，例如当你认为使用[不支持的存储类型](unsupported-storage-drivers.md)的好处大于使用未经测试的功能的风险时，你可能想要启用实验功能。为了让你能够试用这些默认关闭的功能，我们引入了功能开关（feature flag）。 + +实验功能可以通过以下三种方式启用： + +- [使用 CLI](#启动-rancher-时启用功能)：在使用 CLI 安装 Rancher 时，使用功能开关默认启用某个功能。 +- [使用 Rancher UI](#使用-rancher-ui-启用功能)：在**设置**页面启用功能。 +- [使用 Rancher API](#使用-rancher-api-启用功能)：安装 Rancher 后启用功能。 + +每个功能均有以下两个值： + +- 默认值：可以通过在命令行使用标志或环境变量进行配置。 +- 设置值：可以通过 Rancher API 或 UI 进行配置。 + +如果没有设置值，Rancher 会使用默认值。 + +设置值是通过 API 设置的，而默认值是通过命令行设置。因此，如果你使用 API 或 UI 启用或禁用某个功能，命令行中设置的值将被覆盖。 + +如果你安装 Rancher 后使用 Rancher API 将功能开关设置为 true，然后在使用命令升级 Rancher 时将功能开关设置为 false，在这种情况下，虽然默认值会是 false，但是该功能依然会被启用，因为它是通过 API 设置的。如果你随后使用 Rancher API 删除设置值（true）并将它设置为 NULL，则默认值（false）将生效。有关详细信息，请参阅[功能开关页面](../../../getting-started/installation-and-upgrade/installation-references/feature-flags.md)。 + +## 启动 Rancher 时启用功能 + +安装 Rancher 时，使用功能开关启用你所需的功能。通过单节点容器安装 Rancher，和在 Kubernetes 集群上安装 Rancher 对应的命令有所不同。 + +### Kubernetes 安装的情况下启用功能 + +:::note + +通过 Rancher API 设置的值会覆盖命令行传入的值。 + +::: + +使用 Helm Chart 安装 Rancher 时，使用 `--set` 选项。下面的示例通过传递功能开关名称（用逗号分隔）来启用两个功能： + +对于 Kubernetes v1.25 或更高版本，使用 Rancher v2.7.2-v2.7.4 时，将 `global.cattle.psp.enabled` 设置为 `false`。对于 Rancher v2.7.5 及更高版本来说，这不是必需的，但你仍然可以手动设置该选项。 + +``` +helm install rancher rancher-latest/rancher \ + --namespace cattle-system \ + --set hostname=rancher.my.org \ + --set 'extraEnv[0].name=CATTLE_FEATURES' + --set 'extraEnv[0].value==true,=true' +``` + +:::note + +如果你安装的是 alpha 版本，Helm 要求你在命令中添加 `--devel` 选项。 + +::: + +### 离线安装的情况下渲染 Helm Chart + +如果你是在离线环境安装 Rancher 的，在使用 Helm 安装 Rancher 之前，你需要添加一个 Helm Chart 仓库并渲染一个 Helm 模板。详情请参见[离线安装文档](../../../getting-started/installation-and-upgrade/other-installation-methods/air-gapped-helm-cli-install/install-rancher-ha.md)。 + +以下是在渲染 Helm 模板时传入功能开关名称的命令示例。下面的示例通过传递功能开关名称（用逗号分隔）来启用两个功能。 + +Helm 命令如下： + +``` +helm install rancher ./rancher-.tgz \ + --namespace cattle-system \ + --set hostname= \ + --set rancherImage=/rancher/rancher \ + --set ingress.tls.source=secret \ + --set systemDefaultRegistry= \ # 设置在 Rancher 中使用的私有镜像仓库 + --set useBundledSystemChart=true # 使用打包的 Rancher System Chart + --set 'extraEnv[0].name=CATTLE_FEATURES' + --set 'extraEnv[0].value==true,=true' +``` + +### Docker 安装的情况下启用功能 + +如果 Rancher 是使用 Docker 安装的，请使用 `--features` 选项。下面的示例通过传递功能开关名称（用逗号分隔）来启用两个功能： + +``` +docker run -d -p 80:80 -p 443:443 \ + --restart=unless-stopped \ + rancher/rancher:rancher-latest \ + --features==true,=true +``` + +## 使用 Rancher UI 启用功能 + +1. 在左上角，单击 **☰ > 全局设置**。 +1. 单击**功能开关**。 +1. 如需启用某个功能，找到该已禁用的功能，并点击**⋮ > 激活**。 + +**结果**：该功能已启用。 + +### 使用 Rancher UI 禁用功能 + +1. 在左上角，单击 **☰ > 全局设置**。 +1. 单击**功能开关**。你将看到实验功能列表。 +1. 如需禁用某个功能，找到该已启用的功能，并点击**⋮ > 停用**。 + +**结果**：该功能已禁用。 + +## 使用 Rancher API 启用功能 + +1. 前往 `/v3/features`。 +1. 在 `data` 中，你会看到一个数组，该数组包含所有能通过功能开关启用的功能。功能的名称在 `id` 字段中。单击要启用的功能的名称。 +1. 在左上角的 **Operations** 下，点击 **Edit**。 +1. 在 **Value** 下拉菜单中，单击 **True**。 +1. 单击 **Show Request**。 +1. 单击 **Send Request**。 +1. 点击 **Close**。 + +**结果**：该功能已启用。 + +### 使用 Rancher API 禁用功能 + +1. 前往 `/v3/features`。 +1. 在 `data` 中，你会看到一个数组，该数组包含所有能通过功能开关启用的功能。功能的名称在 `id` 字段中。单击要启用的功能的名称。 +1. 在左上角的 **Operations** 下，点击 **Edit**。 +1. 在 **Value** 下拉菜单中，单击 **False**。 +1. 单击 **Show Request**。 +1. 单击 **Send Request**。 +1. 点击 **Close**。 + +**结果**：该功能已禁用。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md new file mode 100644 index 00000000000..e6b3e08e753 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md @@ -0,0 +1,34 @@ +--- +title: 设置指南 +--- + + + + + +本文介绍如何启用 Istio 并在你的项目中使用它。 + +如果你使用 Istio 进行流量管理，则需要允许外部流量进入集群。在这种情况下，你将需要执行以下所有步骤。 + +## 先决条件 + +本指南假设你已经[安装 Rancher](../../../getting-started/installation-and-upgrade/installation-and-upgrade.md)，且已经[配置了一个单独的 Kubernetes 集群](../../new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md)并要在该集群上安装 Istio。 + +集群中的节点必须满足 [CPU 和内存要求](../../../integrations-in-rancher/istio/cpu-and-memory-allocations.md)。 + +Istio 控制的工作负载和服务必须满足 [Istio 要求](https://istio.io/docs/setup/additional-setup/requirements/)。 + +## 安装 + +:::tip 快速设置提示: + +如果你不需要外部流量到达 Istio，而只想设置 Istio 以监控和跟踪集群内的流量，请跳过[设置 Istio Gateway](set-up-istio-gateway.md)和[设置 Istio 的流量管理组件](set-up-traffic-management.md)步骤。 + +::: + +1. [在集群中启用 Istio。](enable-istio-in-cluster.md) +2. [在命名空间中启用 Istio。](enable-istio-in-namespace.md) +3. [使用 Istio Sidecar 添加部署和服务。](use-istio-sidecar.md) +4. [设置 Istio Gateway。](set-up-istio-gateway.md) +5. [设置 Istio 的流量管理组件。](set-up-traffic-management.md) +6. [生成和查看流量。](generate-and-view-traffic.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md new file mode 100644 index 00000000000..ce57b2db349 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md @@ -0,0 +1,50 @@ +--- +title: 项目资源配额 +--- + + + + + +如果多个团队共享一个集群，某个团队可能会使用过多的可用资源，例如 CPU、内存、存储、服务、Kubernetes 对象（如 Pod 或 Secret）等。你可以应用 _资源配额_ 来防止过度消耗资源。资源配额是 Rancher 用来限制项目或命名空间可用资源的功能。 + +本文介绍如何在现有项目中创建资源配额。 + +你也可以在创建新项目时设置资源配额。有关详细信息，请参阅[创建新项目](../../../new-user-guides/manage-clusters/projects-and-namespaces.md#创建项目)。 + +Rancher 中的资源配额包含与 [Kubernetes 原生版本](https://kubernetes.io/docs/concepts/policy/resource-quotas/)相同的功能。Rancher 还扩展了资源配额的功能，从而让你将资源配额应用于项目。有关资源配额如何与 Rancher 中的项目一起使用的详细信息，请参阅[此页面](about-project-resource-quotas.md)。 + +### 将资源配额应用于现有项目 + +修改资源配额的使用场景如下： + +- 限制某个项目和项目下的命名空间能使用的资源 +- 在资源配额已生效的情况下，对项目可用的资源进行扩容或缩容 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面，进入要应用资源配额的集群，然后单击 **Explore**。 +1. 单击**集群 > 项目/命名空间**。 +1. 确保 **Projects/Namespaces** 页面处于 **Group by Project** 视图模式。 + ![Screenshot highlighting the "Group by Project" icon, above the list of projects. It resembles a folder.](/img/edit-project-config-for-resource-quotas-group-by-project.png) + +1. 找到要添加资源配额的项目，选择与项目名称同行的 **⋮**。 + ![Screenshot highlighting triple dots icon at the end of the same row as the project name.](/img/edit-project-config-for-resource-quotas-dots.png) + +1. 选择**编辑配置**。 + +1. 展开**资源限额**并单击**添加资源**。你也可以编辑现有配额。 + +1. 选择资源类型。有关类型的更多信息，请参阅[配额类型参考](resource-quota-types.md)。 + +1. 输入**项目限制**和**命名空间默认限制**的值。 + + | 字段 | 描述 | + | ----------------------- | -------------------------------------------------------------------------------------------------------- | + | 项目限制 | 项目的总资源限制。 | + | 命名空间默认限制 | 每个命名空间的默认资源限制。此限制会沿用到项目中的每个命名空间。项目中所有命名空间的限制之和不应超过项目限制。 | + +1. **可选**：添加更多配额。 + +1. 单击**创建**。 + +**结果**：资源配额已应用到你的项目和命名空间。如果你后续需要添加更多命名空间，Rancher 会验证项目是否可以容纳该命名空间。如果项目无法分配资源，你仍然可以创建命名空间，但命名空间将获得的资源配额为 0。然后 Rancher 将不允许你创建任何受此配额限制的资源。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md new file mode 100644 index 00000000000..1ce6b33f141 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md @@ -0,0 +1,41 @@ +--- +title: 项目管理 +--- + + + + + +_项目_ 是 Rancher 中引入的对象，可帮助你更有组织地管理 Kubernetes 集群中的命名空间。你可以使用项目创建多租户集群，这种集群允许一组用户共享相同的底层资源来创建应用，而应用之间不会相互影响。 + +在层次结构方面： + +- 集群包含项目 +- 项目包含命名空间 + +在 Rancher 中，你可以使用项目将多个命名空间作为一个实体进行管理。在原生 Kubernetes（没有项目这个概念）中，RBAC 或集群资源等功能被分配给了各个命名空间。如果集群中的多个命名空间需要分配同样的访问权限，分配权限会变得非常繁琐。即使所有命名空间都需要相同的权限，但也无法使用一个操作中将这些权限应用于所有命名空间。你必须重复地将这些权限分配给每个命名空间。 + +而 Rancher 通过引入项目的概念，通过允许你在项目级别应用资源和访问权限。然后，项目中的每个命名空间都会继承这些资源和策略。因此你只需将资源和策略分配给项目即可，不需要将它们分配给每个单独的命名空间。 + +你可以使用项目执行以下操作： + +- [为用户分配一组命名空间的访问权限](../../new-user-guides/add-users-to-projects.md) +- 为用户分配[项目中的特定角色](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色)。角色可以是所有者、成员、只读或[自定义](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/custom-roles.md) +- [设置资源配额](manage-project-resource-quotas/manage-project-resource-quotas.md) +- [管理命名空间](../../new-user-guides/manage-namespaces.md) +- [配置工具](../../../reference-guides/rancher-project-tools.md) +- [配置 Pod 安全策略](manage-pod-security-policies.md) + +### 授权 + +非管理者用户只有在[管理员](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md)、[集群所有者或成员](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#集群角色)或[项目所有者](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色)将非管理员用户添加到项目的**成员**选项卡后，才能获取项目的访问权限。 + +创建项目的人自动成为[项目所有者](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色)。 + +## 在项目之间切换 + +要在项目之间切换，请使用导航栏中的下拉菜单。你也可以直接在导航栏中切换项目： + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面，进入要切换项目的集群然后点击 **Explore**。 +1. 在顶部导航栏中，选择要打开的项目。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/monitoring-alerting-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/monitoring-alerting-guides.md new file mode 100644 index 00000000000..16a504ae226 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/monitoring-alerting-guides.md @@ -0,0 +1,15 @@ +--- +title: Monitoring/Alerting 指南 +--- + + + + + +- [启用 Monitoring](enable-monitoring.md) +- [卸载 Monitoring](uninstall-monitoring.md) +- [为工作负载设置 Monitoring](set-up-monitoring-for-workloads.md) +- [自定义 Grafana 仪表板](customize-grafana-dashboard.md) +- [持久化 Grafana 仪表板](create-persistent-grafana-dashboard.md) +- [调试高内存用量](debug-high-memory-usage.md) + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/enable-prometheus-federator.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/enable-prometheus-federator.md index bf3b78cac72..c6bac839971 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/enable-prometheus-federator.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/enable-prometheus-federator.md @@ -4,7 +4,7 @@ title: 启用 Prometheus Federator ## 要求 -默认情况下，Prometheus Federator 已配置并旨在与 [rancher-monitoring](https://rancher.com/docs/rancher/v2.6/en/monitoring-alerting/) 一起部署。rancher-monitoring 同时部署了 Prometheus Operator 和 Cluster Prometheus，每个项目监控堆栈（Project Monitoring Stack）默认会联合命名空间范围的指标。 +默认情况下，Prometheus Federator 已配置并旨在与 [rancher-monitoring](../../../../pages-for-subheaders/monitoring-and-alerting.md) 一起部署。rancher-monitoring 同时部署了 Prometheus Operator 和 Cluster Prometheus，每个项目监控堆栈（Project Monitoring Stack）默认会联合命名空间范围的指标。 有关安装 rancher-monitoring 的说明，请参阅[此页面](../enable-monitoring.md)。 @@ -75,7 +75,7 @@ matchLabels: 1. 单击 **Prometheus Federator** Chart。 1. 单击**安装**。 1. 在**元数据**页面，点击**下一步**。 -1. 在**项目 Release 命名空间项目 ID** 字段中，`System 项目`是默认值，但你可以使用具有类似[有限访问权限](#确保-cattle-monitoring-system-命名空间位于-system-项目中或者位于一个锁定并能访问集群中其他项目的项目中)的另一个项目覆盖它。你可以在 local 上游集群中运行以下命令来找到项目 ID： +1. 在**命名空间** > **项目 Release 命名空间项目 ID** 字段中，`System 项目`是默认值，但你可以使用具有类似[有限访问权限](#确保-cattle-monitoring-system-命名空间位于-system-项目中或者位于一个锁定并能访问集群中其他项目的项目中)的另一个项目覆盖它。你可以在 local 上游集群中运行以下命令来找到项目 ID： ```plain kubectl get projects -A -o custom-columns="NAMESPACE":.metadata.namespace,"ID":.metadata.name,"NAME":.spec.displayName diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/prometheus-federator-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/prometheus-federator-guides.md new file mode 100644 index 00000000000..7bc98a902e4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/prometheus-federator-guides/prometheus-federator-guides.md @@ -0,0 +1,12 @@ +--- +title: Prometheus Federator 指南 +--- + + + + + +- [启用 Prometheus Federator](enable-prometheus-federator.md) +- [卸载 Prometheus Federator](uninstall-prometheus-federator.md) +- [自定义 Grafana 仪表板](customize-grafana-dashboards.md) +- [为工作负载设置 Prometheus Federator](set-up-workloads.md) \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/set-up-monitoring-for-workloads.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/set-up-monitoring-for-workloads.md index 9b8fd1582c8..7ab5dd36f79 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/set-up-monitoring-for-workloads.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-alerting-guides/set-up-monitoring-for-workloads.md @@ -8,7 +8,7 @@ title: 为工作负载设置 Monitoring 如果你只需要工作负载的 CPU 和内存时间序列，则不需要部署 ServiceMonitor 或 PodMonitor，因为 Monitoring 应用默认会收集资源使用情况的指标数据。资源使用的时间序列数据在 Prometheus 的本地时间序列数据库中。 -Grafana 显示聚合数据，你也可以使用 PromQL 查询来查看单个工作负载的数据。进行 PromQL 查询后，你可以在 Prometheus UI 中单独执行查询并查看可视化的时间序列，你也可以使用查询来自定义显示工作负载指标的 Grafana 仪表板。有关工作负载指标的 PromQL 查询示例，请参阅[本节](https://rancher.com/docs/rancher/v2.6/en/monitoring-alerting/expression/#workload-metrics)。 +Grafana 显示聚合数据，你也可以使用 PromQL 查询来查看单个工作负载的数据。进行 PromQL 查询后，你可以在 Prometheus UI 中单独执行查询并查看可视化的时间序列，你也可以使用查询来自定义显示工作负载指标的 Grafana 仪表板。有关工作负载指标的 PromQL 查询示例，请参阅[本节](../../../integrations-in-rancher/monitoring-and-alerting/promql-expressions.md#工作负载指标)。 要为你的工作负载设置自定义指标，你需要设置一个 Exporter 并创建一个新的 ServiceMonitor 自定义资源，从而将 Prometheus 配置为从 Exporter 中抓取指标。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/advanced-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/advanced-configuration.md new file mode 100644 index 00000000000..549bc6fa852 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/advanced-configuration.md @@ -0,0 +1,19 @@ +--- +title: 高级配置 +--- + + + + + +### Alertmanager + +有关配置 Alertmanager 自定义资源的信息，请参阅[此页面。](alertmanager.md) + +### Prometheus + +有关配置 Prometheus 自定义资源的信息，请参阅[此页面。](prometheus.md) + +### PrometheusRules + +有关配置 PrometheusRules 自定义资源的信息，请参阅[此页面。](prometheusrules.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md index fc12de7e488..458011a703c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md @@ -1,5 +1,5 @@ --- -title: 配置 PrometheusRule +title: PrometheusRule 配置 --- PrometheusRule 定义了一组 Prometheus 告警和/或记录规则。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md new file mode 100644 index 00000000000..3b53fe7214f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/monitoring-v2-configuration-guides.md @@ -0,0 +1,55 @@ +--- +title: 配置 +--- + + + + + +本文介绍在 Rancher UI 中配置 Monitoring V2 的一些最重要选项。 + +有关为 Prometheus 配置自定义抓取目标和规则的信息，请参阅 [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) 的上游文档。Prometheus Operator [设计文档](https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/design.md)中解释了一些最重要的自定义资源。Prometheus Operator 文档还可以帮助你设置 RBAC、Thanos 或进行自定义配置。 + +## 设置资源限制和请求 + +安装 `rancher-monitoring` 时可以配置 Monitoring 应用的资源请求和限制。有关默认限制的更多信息，请参阅[此页面](../../../reference-guides/monitoring-v2-configuration/helm-chart-options.md#配置资源限制和请求)。 + +:::tip + +在空闲集群上，Monitoring 可能会占用很多 CPU 资源。要提高性能，请关闭 Prometheus Adapter。 + +::: + +## Prometheus 配置 + +通常不需要直接编辑 Prometheus 自定义资源。 + +相反，要让 Prometheus 抓取自定义指标，你只需创建一个新的 ServiceMonitor 或 PodMonitor 来将 Prometheus 配置为抓取其他指标。 + +### ServiceMonitor 和 PodMonitor 配置 + +有关详细信息，请参阅[此页面](../../../reference-guides/monitoring-v2-configuration/servicemonitors-and-podmonitors.md)。 + +### 高级 Prometheus 配置 + +有关直接编辑 Prometheus 自定义资源（对高级用例可能有帮助）的更多信息，请参阅[此页面](advanced-configuration/prometheus.md)。 + +## Alertmanager 配置 + +Alertmanager 自定义资源通常不需要直接编辑。在常见用例中，你可以通过更新路由和接收器来管理告警。 + +路由和接收器是 Alertmanager 自定义资源配置的一部分。在 Rancher UI 中，路由（Route）和接收器（Receiver）并不是真正的自定义资源，而是 Prometheus Operator 用来将你的配置与 Alertmanager 自定义资源同步的伪自定义资源。当路由和接收器更新时，Monitoring 应用将自动更新 Alertmanager 来反映这些更改。 + +对于一些高级用例，你可能需要直接配置 Alertmanager。有关详细信息，请参阅[此页面](advanced-configuration/alertmanager.md)。 + +### 接收器 + +接收器（Receiver）用于设置通知。有关如何配置接收器的详细信息，请参阅[此页面](../../../reference-guides/monitoring-v2-configuration/receivers.md)。 + +### 路由 + +路由（Route）在通知到达接收器之前过滤它们。每条路由都需要引用一个已经配置好的接收器。有关如何配置路由的详细信息，请参阅[此页面](../../../reference-guides/monitoring-v2-configuration/routes.md)。 + +### 高级配置 + +有关直接编辑 Alertmanager 自定义资源（对高级用例可能有帮助）的更多信息，请参阅[此页面](advanced-configuration/alertmanager.md)。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/about-provisioning-drivers.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/about-provisioning-drivers.md new file mode 100644 index 00000000000..94b0333f481 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/about-provisioning-drivers.md @@ -0,0 +1,51 @@ +--- +title: 配置驱动 +--- + + + + + +使用 Rancher 中的驱动，你可以管理可以使用哪些供应商来部署[托管的 Kubernetes 集群](../../kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md)或[云服务器节点](../../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)，以允许 Rancher 部署和管理 Kubernetes。 + +### Rancher 驱动 + +你可以启用或禁用 Rancher 中内置的驱动。如果相关驱动 Rancher 尚未实现，你可以添加自己的驱动。 + +Rancher 中有两种类型的驱动： + +* [集群驱动](#集群驱动) +* [主机驱动](#主机驱动) + +### 集群驱动 + +集群驱动用于配置[托管的 Kubernetes 集群](../../kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md)，例如 GKE、EKS、AKS 等。创建集群时可以显示的集群驱动，是由集群驱动的状态定义的。只有 `active` 集群驱动将显示为为托管 Kubernetes 集群创建集群的选项。默认情况下，Rancher 与几个现有的集群驱动打包在一起，但你也可以创建自定义集群驱动并添加到 Rancher。 + +默认情况下，Rancher 已激活多个托管 Kubernetes 云提供商，包括： + +* [Amazon EKS](../../kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/eks.md) +* [Google GKE](../../kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/gke.md) +* [Azure AKS](../../kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/aks.md) + +还有几个托管的 Kubernetes 云提供商是默认禁用的，但也打包在 Rancher 中： + +* [Alibaba ACK](../../kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/alibaba.md) +* [Huawei CCE](../../kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/huawei.md) +* [Tencent](../../kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/tencent.md) + +### 主机驱动 + +主机驱动用于配置主机，Rancher 使用这些主机启动和管理 Kubernetes 集群。主机驱动与 [Docker Machine 驱动](https://docs.docker.com/machine/drivers/)相同。创建主机模板时可以显示的主机驱动，是由主机驱动的状态定义的。只有 `active` 主机驱动将显示为创建节点模板的选项。默认情况下，Rancher 与许多现有的 Docker Machine 驱动打包在一起，但你也可以创建自定义主机驱动并添加到 Rancher。 + +如果你不想向用户显示特定的主机驱动，则需要停用这些主机驱动。 + +Rancher 支持几家主要的云提供商，但默认情况下，这些主机驱动处于 active 状态并可供部署： + +* [Amazon EC2](../../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-amazon-ec2-cluster.md) +* [Azure](../../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster.md) +* [Digital Ocean](../../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-a-digitalocean-cluster.md) +* [vSphere](../../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md) + +还有其他几个默认禁用的主机驱动，但打包在 Rancher 中： + +* [Harvester](../../../../integrations-in-rancher/harvester/overview.md#harvester-主机驱动) - 在 Rancher 2.6.1 中可用 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md new file mode 100644 index 00000000000..de9bc25d31a --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-rke1-templates/about-rke1-templates.md @@ -0,0 +1,130 @@ +--- +title: RKE 模板 +--- + + + + + +RKE 模板旨在让 DevOps 和安全团队标准化和简化 Kubernetes 集群创建的流程。 + +RKE 的全称是 [Rancher Kubernetes Engine](https://rancher.com/docs/rke/latest/en/)，它是 Rancher 用来配置 Kubernetes 集群的工具。 + +随着 Kubernetes 越来越受欢迎，管理更多小型集群逐渐成为趋势。如果你想要创建大量集群，对集群进行一致管理尤为重要。多集群管理面临着安全和附件配置执行的挑战，在将集群移交给最终用户之前，这些配置需要标准化。 + +RKE 模板有助于标准化这些配置。无论是使用 Rancher UI、Rancher API 还是自动化流程创建的集群，Rancher 都将保证从 RKE 集群模板创建的每个集群在生成方式上是一致的。 + +管理员可以控制最终用户能更改的集群选项。RKE 模板还可以与特定的用户和组共享，以便管理员可以为不同的用户集创建不同的 RKE 模板。 + +如果集群是使用 RKE 模板创建的，则不能让集群使用另一个 RKE 模板。你只能将集群更新为同一模板的新版本。 + +你可以[将现有集群的配置保存为 RKE 模板](apply-templates.md#将现有集群转换为使用-rke-模板)。这样，只有模板更新后才能更改集群的设置。新模板还可用于启动新集群。 + +RKE 模板的核心功能允许 DevOps 和安全团队： + +- 标准化集群配置并确保按照最佳实践创建 Rancher 配置的集群 +- 配置集群时，防止用户做出不明智的选择 +- 与不同的用户和组共享不同的模板 +- 将模板的所有权委托给受信任的用户进行更改 +- 控制哪些用户可以创建模板 +- 要求用户使用模板来创建集群 + +## 可配置的设置 + +RKE 模板可以在 Rancher UI 中创建或以 YAML 格式定义。当你使用 Rancher 从基础设施提供商配置自定义节点或一般节点时，它们可以指定为相同的参数： + +- 云提供商选项 +- Pod 安全选项 +- 网络提供商 +- Ingress Controller +- 网络安全配置 +- 网络插件 +- 私有镜像仓库 URL 和凭证 +- 附加组件 +- Kubernetes 选项，包括 kube-api、kube-controller、kubelet 和服务等 Kubernetes 组件的配置 + +RKE 模板的[附加组件](#附加组件)的功能特别强大，因为它允许多种自定义选项。 + +## RKE 模板的范围 + +Rancher 配置的集群支持 RKE 模板。模板可用于配置自定义集群或由基础设施提供商启动的集群。 + +RKE 模板用于定义 Kubernetes 和 Rancher 设置。节点模板负责配置节点。有关如何将 RKE 模板与硬件结合使用的参考，请参阅 [RKE 模板和硬件](infrastructure.md)。 + +可以从头开始创建 RKE 模板来预先定义集群配置。它们可以用于启动新集群，也可以从现有的 RKE 集群导出模板。 + +现有集群的设置可以[保存为 RKE 模板](apply-templates.md#将现有集群转换为使用-rke-模板)。这会创建一个新模板并将集群设置绑定到该模板。这样，集群只有在[模板更新](manage-rke1-templates.md#更新模板)的情况下才能[使用新版本的模板](manage-rke1-templates.md#升级集群以使用新的模板修订版)进行升级。新模板也可以用来创建新集群。 + + +## 示例场景 +如果一个组织同时拥有普通和高级 Rancher 用户，管理员可能希望为高级用户提供更多用于集群创建的选项，并限制普通用户的选项。 + +这些[示例场景](example-use-cases.md)描述组织如何使用模板来标准化集群创建。 + +示例场景包括： + +- **强制执行模板**：如果希望所有 Rancher 配置的新集群都具有某些设置，管理员可能想要[为每个用户强制执行一项或多项模板设置](example-use-cases.md#强制执行模板设置)。 +- **与不同的用户共享不同的模板**：管理员可以为[普通用户和高级用户提供不同的模板](example-use-cases.md#普通用户和高级用户模板)。这样，普通用户会有更多限制选项，而高级用户在创建集群时可以使用更多选项。 +- **更新模板设置**：如果组织的安全和 DevOps 团队决定将最佳实践嵌入到新集群所需的设置中，这些最佳实践可能会随着时间而改变。如果最佳实践发生变化，[可以将模板更新为新版本](example-use-cases.md#更新模板和集群)，这样，使用模板创建的集群可以[升级到模板的新版本](manage-rke1-templates.md#升级集群以使用新的模板修订版)。 +- **共享模板的所有权**：当模板所有者不再想要维护模板或想要共享模板的所有权时，此方案描述了如何[共享模板所有权](example-use-cases.md#允许其他用户控制和共享模板)。 + +## 模板管理 + +创建 RKE 模板时，可以在 Rancher UI 中的**集群管理**下的 **RKE 模板**中使用模板。创建模板后，你将成为模板所有者，这将授予你修改和共享模板的权限。你可以与特定用户或组共享 RKE 模板，也可以公开模板。 + +管理员可以开启模板强制执行，要求用户在创建集群时始终使用 RKE 模板。这使管理员可以保证 Rancher 总是创建指定配置的集群。 + +RKE 模板更新通过修订系统处理。如果要更改或更新模板，请创建模板的新版本。然后，可以将使用旧版本模板创建的集群升级到新模板修订版。 + +在 RKE 模板中，模板所有者可以限制设置的内容，也可以打开设置以供最终用户选择值。它们的差别体现在，创建模板时，Rancher UI 中的每个设置上的**允许用户覆盖**标示。 + +对于无法覆盖的设置，最终用户将无法直接编辑它们。为了让用户使用这些设置的不同选项，RKE 模板所有者需要创建 RKE 模板的新版本，这将允许用户升级和更改该选项。 + +本节中的文件解释了 RKE 模板管理的细节： + +- [获取创建模板的权限](creator-permissions.md) +- [创建和修改模板](manage-rke1-templates.md) +- [强制执行模板设置](enforce-templates.md#强制新集群使用-rke-模板) +- [覆盖模板设置](override-template-settings.md) +- [与集群创建者共享模板](access-or-share-templates.md#与特定用户或组共享模板) +- [共享模板的所有权](access-or-share-templates.md#共享模板所有权) + +你可以参见此[模板的示例 YAML 文件](../../../../reference-guides/rke1-template-example-yaml.md)作为参考。 + +## 应用模板 + +你可以使用你自己创建的模板来[创建集群](apply-templates.md#使用-rke-模板创建集群)，也可以使用[与你共享的模板](access-or-share-templates.md)来创建集群。 + +如果 RKE 模板所有者创建了模板的新版本，你可以[将你的集群升级到该版本](apply-templates.md#更新使用-rke-模板创建的集群)。 + +可以从头开始创建 RKE 模板来预先定义集群配置。它们可以用于启动新集群，也可以从现有的 RKE 集群导出模板。 + +你可以[将现有集群的配置保存为 RKE 模板](apply-templates.md#将现有集群转换为使用-rke-模板)。这样，只有模板更新后才能更改集群的设置。 + +## 标准化硬件 + +RKE 模板的目的是标准化 Kubernetes 和 Rancher 设置。如果你还想标准化你的基础设施，一个选择是将 RKE 模板与[其他工具](infrastructure.md)一起使用。 + +另一种选择是使用包含节点池配置选项，但不强制执行配置的[集群模板](../../manage-clusters/manage-cluster-templates.md)。 + +## YAML 定制 + +如果将 RKE 模板定义为 YAML 文件，则可以修改此[示例 RKE 模板 YAML](../../../../reference-guides/rke1-template-example-yaml.md)。RKE 模板中的 YAML 使用了 Rancher 在创建 RKE 集群时使用的相同自定义设置。但由于 YAML 要在 Rancher 配置的集群中使用，因此需要将 RKE 模板自定义项嵌套在 YAML 中的 `rancher_kubernetes_engine_config` 参数下。 + +RKE 文档也提供[注释的](https://rancher.com/docs/rke/latest/en/example-yamls/) `cluster.yml` 文件供你参考。 + +有关可用选项的更多信息，请参阅[集群配置](https://rancher.com/docs/rke/latest/en/config-options/)上的 RKE 文档。 + +### 附加组件 + +RKE 模板配置文件的附加组件部分的工作方式与[集群配置文件的附加组件部分](https://rancher.com/docs/rke/latest/en/config-options/add-ons/)相同。 + +用户定义的附加组件指令允许你调用和下拉 Kubernetes 清单或将它们直接内联。如果这些 YAML 清单包括在 RKE 模板中，Rancher 将在集群中部署这些 YAML 文件。 + +你可以使用附加组件执行以下操作： + +- 启动 Kubernetes 集群后，在集群上安装应用 +- 在使用 Kubernetes Daemonset 部署的节点上安装插件 +- 自动设置命名空间、ServiceAccount 或角色绑定 + +RKE 模板配置必须嵌套在 `rancher_kubernetes_engine_config` 参数中。要设置附加组件，在创建模板时单击**以 YAML 文件编辑**。然后使用 `addons` 指令添加清单，或使用 `addons_include` 指令设置哪些 YAML 文件可用于附加组件。有关自定义附加组件的更多信息，请参见[用户自定义附加组件文档](https://rancher.com/docs/rke/latest/en/config-options/add-ons/user-defined-add-ons/)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md new file mode 100644 index 00000000000..b9997d6e7b4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md @@ -0,0 +1,145 @@ +--- +title: 配置认证 +weight: 10 +--- + + + + + +Rancher 为 Kubernetes 添加的一个关键功能是集中式用户认证，这个特性允许用户使用一组凭证对任何 Kubernetes 集群进行身份认证。 + +这种集中式用户认证是通过 Rancher 的认证代理完成的，该代理与 Rancher 的其余部分一并安装，此代理对用户进行认证并通过一个 Service Acount 将请求转发到 Kubernetes 集群中。 + +:::warning + +用来启用外部认证的账户将被授予管理员权限。如果你使用一个测试账号或非管理员账号，该账号仍然会被授予管理员级别权限。请查看[外部认证配置和主体用户](#外部认证配置和用户主体)了解原因。 + +::: + +## 外部认证与本地认证 + +Rancher 认证代理可以与以下外部认证服务集成。 + +| 认证服务 | +| ---------------------------------------------------------------------------------------------------------------------- | +| [Microsoft Active Directory](configure-active-directory.md) | +| [GitHub](configure-github.md) | +| [Microsoft Azure AD](configure-azure-ad.md) | +| [FreeIPA](configure-freeipa.md) | +| [OpenLDAP](../configure-openldap/configure-openldap.md) | +| [Microsoft AD FS](../configure-microsoft-ad-federation-service-saml/configure-microsoft-ad-federation-service-saml.md) | +| [PingIdentity](configure-pingidentity.md) | +| [Keycloak (OIDC)](configure-keycloak-oidc.md) | +| [Keycloak (SAML)](configure-keycloak-saml.md) | +| [Okta](configure-okta-saml.md) | +| [Google OAuth](configure-google-oauth.md) | +| [Shibboleth](../configure-shibboleth-saml/configure-shibboleth-saml.md) | + +当然，Rancher 也提供[本地认证](create-local-users.md). + +在多数情况下，你应该使用外部认证服务而不是使用本地认证，因为外部认证服务可以集中式的对用户进行管理。但是在极少数情况下，例如外部认证服务不可用或正在维护时，你可能需要使用本地认证用户来管理 Rancher。 + +## 用户和组 + +Rancher 依赖用户和组来决定允许谁登录 Rancher 以及他们可以访问哪些资源。当使用外部认证时，外部认证系统会根据用户提供组的信息。这些用户和组被赋予了集群、项目、多集群应用以及全局 DNS 提供商和条目等资源的特定角色。当你对组进行授权时，在认证服务中所有属于这个组中的用户都有访问指定的资源的权限。有关角色和权限的更多信息，请查看 [RBAC](../manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md)。 + +:::note + +本地认证不支持创建或管理组 + +::: + +更多信息，请查看[用户和组](manage-users-and-groups.md) + +## Rancher 授权范围 + +当你配置完 Rancher 使用外部认证服务后，你可以配置允许谁登录和使用 Rancher，包含如下的选项： + +| 访问级别 | 描述 | +| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | +| 允许任何有效用户 | 在认证服务中的*任何*用户都可以访问 Rancher。通常情况下不建议使用该设置！ | +| 允许集群和项目成员，以及授权的用户和组织 | 认证服务中属于**集群成员**或**项目成员**的用户或组成员都可以登录 Rancher。此外添加在**授权的用户和组织**列表中的用户和组成员也可以登录到 Rancher。 | +| 仅限于授权的用户可以访问 | 仅有在授权用户和组织列表中的用户和组成员可以登录到 Rancher。 | + +要在授权服务中为用户设置 Rancher 访问级别，请执行以下步骤： + +1. 在左上角，点击 **☰ > 用户 & 认证**。 +1. 在左侧导航栏，点击 **认证**. +1. 设置完外部认证详细信息后，使用 **站点访问** 选项配置用户权限范围，上面的表格说明了每个选项的访问级别。 +1. 可选：如果你选择 **允许任何有效用户** 以外的选项，你可以通过在出现的文本框中搜索用户，将用户添加到授权用户和组织的列表中。 +1. 点击 **保存**。 + +**结果:** Rancher 的访问配置被应用。 + +:::note SAML 认证警告: + +- SAML 协议不支持搜索或查找用户或组。因此，将用户或组添加到 Rancher 时不会对其进行验证。 +- 添加用户时，必须正确输入确切的用户 ID（即 UID 字段）。键入用户 ID 时，将不会搜索可能匹配的其他用户 ID。 +- 添加组时，必须从文本框旁边的下拉列表中选择组。Rancher 假定来自文本框的任何输入都是用户。 +- 用户组下拉列表仅显示您所属的用户组。您将无法添加您不是其成员的组。 + +::: + +## 外部认证配置和用户主体 + +配置外部认证需要： + +- 分配了管理员角色的本地用户，以下称为 _本地主体_。 +- 可以使用外部认证服务进行认证的外部用户，以下简称为 _外部主体_。 + +外部认证的配置也会影响 Rancher 中主体用户的管理方式，具体地说，当用户账户启用了外部认证时，将授予其管理员级别的权限。这是因为本地主体和外部主体共享相同的用户 ID 和访问权限。 + +以下说明演示了这些效果： + +1. 作为本地主体登录到 Rancher 并完成外部身份验证的配置。 + + ![Sign In](/img/sign-in.png) + +2. Rancher 将外部主体与本地主体相关联。这两个用户共享本地主体的用户 ID。 + + ![Principal ID Sharing](/img/principal-ID.png) + +3. 完成配置后，Rancher 将自动退出本地主体。 + + ![Sign Out Local Principal](/img/sign-out-local.png) + +4. 然后，Rancher 会自动将您登录外部主体。 + + ![Sign In External Principal](/img/sign-in-external.png) + +5. 因为外部主体和本地主体共享一个 ID，所以用户列中不会再单独显示一个另外的外部主体的对象。 + + ![Sign In External Principal](/img/users-page.png) + +6. 外部主体和本地主体共享相同的访问权限。 + +:::note 重新配置先前设置的认证 + +如果需要重新配置或禁用后重新启用先前设置过的认证，请确保尝试这样做的用户以外部用户身份登录到 Rancher，而不是使用本地管理员登录。 + +::: + +## 禁用认证 + +当你禁用认证时，Rancher 会删除所有与之关联的资源，例如： + +- 密文 +- 绑定的全局角色。 +- 绑定的集群角色。 +- 绑定的项目角色。 +- 与外部认证关联但从未以本地用户身份登录 Rancher 的外部用户。 + +由于此操作可能会导致许多资源丢失，因此你可能需要添加一些保护措施。若要确保禁用外部认证时不执行清理流程，需要为外部认证的配置添加特殊的注释。 + +例如，若要对 Azure AD 认证增加保护措施，你需要在 authconfig 对象上增加 `azuread` 注释： + +`kubectl annotate --overwrite authconfig azuread management.cattle.io/auth-provider-cleanup='user-locked'` + +禁用 Azure AD 认证后，Rancher 不会执行清理流程，直到你将该注解设置为 `unlocked`。 + +### 手动运行资源清理 + +Rancher 可能会在本地集群中保留之前禁用的外部认证配置的资源，即使你配置对接了另一种认证也是如此。例如，如果你对接了 A 认证，然后禁用它，并重新对接使用 B 认证，当你升级到新版本的 Rancher 时，你可以手动触发对认证 A 配置的资源清理。 + +要手动触发已禁用的认证配置的清理，请将 `unlocked` 值添加到对应认证配置的 `management.cattle.io/auth-provider-cleanup` 注解中。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/configure-okta-saml.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/configure-okta-saml.md index 57cbaf95418..212f7cf45d1 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/configure-okta-saml.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/configure-okta-saml.md @@ -104,4 +104,4 @@ OpenLDAP ServiceAccount 用于所有搜索。无论用户个人的 SAML 权限 1. 单击 **Okta**，如果已配置 SAML，则单击**编辑配置**。 1. 在**用户和组搜索**下，选中**配置 OpenLDAP Server**。 -如果你在测试与 OpenLDAP Server 的连接时遇到问题，请确保你输入了ServiceAccount 的凭证并正确配置了搜索库。你可以检查 Rancher 日志来查明根本原因。调试日志可能包含有关错误的更详细信息。请参阅[如何启用调试日志](../../../../faq/technical-items.md#如何启用调试日志记录)了解更多信息。 \ No newline at end of file +如果你在测试与 OpenLDAP Server 的连接时遇到问题，请确保你输入了ServiceAccount 的凭证并正确配置了搜索库。你可以检查 Rancher 日志来查明根本原因。调试日志可能包含有关错误的更详细信息。请参阅[如何启用调试日志](../../../../faq/technical-items.md#如何启用调试日志记录)了解更多信息。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-permissions-and-global-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-permissions-and-global-configuration.md new file mode 100644 index 00000000000..5838ed66fed --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-permissions-and-global-configuration.md @@ -0,0 +1,85 @@ +--- +title: 认证、权限和全局配置 +--- + + + + + +安装完成后，[系统管理员](manage-role-based-access-control-rbac/global-permissions.md) 应该通过 Rancher 配置认证、授权、安全性、默认设置、安全策略、驱动和全局 DNS 条目。 + +## 首次登录 + +首次登录 Rancher 后，Rancher 会提示你输入 **Rancher Server URL**，你应该将 URL 设置为访问 Rancher Server 的主入口点。当负载均衡器运行在 Rancher Server 集群前面时，URL 应该设置为负载均衡地址。系统会自动尝试根据运行 Rancher Server 的主机 IP 地址或主机名推断 Rancher Server URL，但只有当 Rancher Server 以单节点方式安装时才有效。因此在大多数情况下，你都需要将 Rancher Server URL 设置为正确的值。 + +:::danger + +当设置完 Rancher Server URL 后，我们不支持修改它。请格外小心的设置此项配置。 + +::: + +## 认证 + +Rancher 为 Kubernetes 增加了一项关键特性是集中式的用户认证。此特性允许设置本地用户和/或连接到外部认证程序。通过连接到外部认证程序，你可以使用该程序提供的用户和组。 + +更多关于认证的工作原理以及如何配置对接各个认证程序，请参考[认证](authentication-config/authentication-config.md)。 + +## 授权 + +在 Rancher 中，每个人都是以 _用户_ 的身份进行鉴权，这是一个授予你访问 Rancher 的登录身份。用户登录 Rancher 后，他们的 _授权_ 或者他们在系统中的访问权限由用户的角色决定。Rancher 提供了内置的角色，允许你你轻松地配置用户对资源的权限，但是 Rancher 还提供了为每个 Kubernetes 资源自定义角色的功能。 + +更多关于授权的工作原理以及自定义角色的使用，请参考 [RBAC](manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md)。 + +## Pod 安全策略 + +_Pod 安全策略_ (或 PSPs) 是控制 Pod 安全敏感方面规范的对象，例如 root 权限。如果一个 Pod 不满足 PSP 中指定的条件，Kubernetes 将不允许 Pod 启动，同时 Rancher 会显示一条错误信息。 + +更多关于如何创建和使用 PSPs 的内容，请参考 [Pod 安全策略](create-pod-security-policies.md)。 + +## Provisioning Drivers + +Rancher 中的驱动允许你管理哪些程序可以预置[托管的 Kubernetes 集群](../kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md) 或 [云服务器节点](../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)，允许 Rancher 部署和管理 Kubernetes。 + +更多信息请参考 [Provisioning Drivers](about-provisioning-drivers/about-provisioning-drivers.md)。 + +## 添加 Kubernetes 版本到 Rancher 中 + +使用此功能，你可以在最新版本的 Kubernetes 发布后立即升级，而不需要升级 Rancher。此功能允许你轻松升级 Kubernetes 的补丁版本（例如 `v1.15.X`），但不打算升级 Kubernetes 的次要版本（例如 `v1.X.0`），因为 Kubernetes 倾向于在次要版本之间弃用或添加 API。 + +Rancher 用于配置 [RKE 集群](../launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) 的信息现在存储于 Rancher Kubernetes 元数据中，更多关于元数据的配置以及如何更改用于配置 RKE 集群的 Kubernetes 版本的信息，请参考 [Rancher Kubernetes 元数据](../../../getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md)。 + +Rancher Kubernetes 元数据包含 Kubernetes 版本信息，Rancher 使用这些信息来配置 [RKE 集群](../launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)。 + +关于元数据的工作原理以及如何配置元数据，请参考 [Rancher Kubernetes 元数据](../../../getting-started/installation-and-upgrade/upgrade-kubernetes-without-upgrading-rancher.md)。 + +## 全局设置 + +控制某些全局级别 Rancher 配置项可以在顶部的导航栏中找到。 + +点击左上角的 **☰** ，然后选择 **全局设置**，查看和配置以下设置： + +- **设置**: 各种 Rancher 默认值，例如用户密码的最小长度 (`password-min-length`)。在修改这些设置项时应该谨慎，因为设置无效的值可能会破坏 Rancher 的安装。 +- **功能开关**: 可以打开或关闭 Rancher 的某些功能，一些标志用于 [实验性功能](#启用实验性功能). +- **横幅**: 可以添加到门户上固定位置的元素，例如你可以使用这些选项在用户登录 Rancher 时为他们设置[自定义的横幅](custom-branding.md#固定横幅)。 +- **品牌**: 你可以[自定义](custom-branding.md) Rancher UI 的设计元素，你可以增加一个自定义的 logo 或 favicon，也可以修改 UI 的颜色。 +- **性能**: Rancher UI 的性能设置，例如增量资源加载。 +- **主页链接**: Rancher UI **主页**页面上显示的链接，你可以修改默认链接的可见性或者增加自己的链接。 + +### 启用实验性功能 + +Rancher 包含一些默认处于实验性和/或禁用的功能，功能开关允许你启用这些特性。更多信息请参考[功能开关](../../advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 + +### 全局配置 + +仅在激活 **legacy** [功能开关](../../advanced-user-guides/enable-experimental-features/enable-experimental-features.md) 时才可以看见**全局配置**选项。在 v2.6 及更新版本新安装的 Rancher 已经默认禁用了 **legacy** 特性。如果你是从早期的 Rancher 版本升级，或者在 Rancher v2.6 及更新版本上启用了 **legacy** 特性，顶部导航菜单中将会显示**全局配置**： + +1. 点击左上角的 **☰**。 +1. 在 **旧版应用** 中选择 **全局配置**。 + +**全局配置**提供以下功能： + +- **应用商店** +- **全局 DNS 条目** +- **全局 DNS 提供商** + +由于这些是旧版特性，请参考 Rancher v2.0-v2.4 的[应用商店](/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md), [全局 DNS 条目](/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/helm-charts-in-rancher/globaldns.md#adding-a-global-dns-entry), 以及 [全局 DNS 提供商](/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/helm-charts-in-rancher/globaldns.md#editing-a-global-dns-provider)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-microsoft-ad-federation-service-saml/configure-microsoft-ad-federation-service-saml.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-microsoft-ad-federation-service-saml/configure-microsoft-ad-federation-service-saml.md new file mode 100644 index 00000000000..3b96b5f62ff --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-microsoft-ad-federation-service-saml/configure-microsoft-ad-federation-service-saml.md @@ -0,0 +1,39 @@ +--- +title: 配置 Microsoft AD FS (SAML) +--- + + + + + +如果你的组织使用 Active Directory Federation Service (AD FS) 进行用户身份认证，你可以通过配置 Rancher 来允许用户使用 AD FS 凭证登录。 + +## 先决条件 + +已安装 Rancher。 + +- 获取你的 Rancher Server URL。配置 AD FS 时，请使用该 URL 替换 `` 占位符。 +- 你必须在 Rancher 安装时具有全局管理员账号。 + +你必须配置 [Microsoft AD FS 服务器](https://docs.microsoft.com/en-us/windows-server/identity/active-directory-federation-services)。 + +- 获取你的 AD FS 服务器 IP/DNS 名称。配置 AD FS 时，请使用该 IP/DNS 名称替换 `` 占位符。 +- 你必须有在 AD FS 服务器上添加 [Relying Party Trusts](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/create-a-relying-party-trust) 的权限。 + +## 配置概要 + +要让 Rancher Server 使用 Microsoft AD FS，你需要在 Active Directory 服务器上配置 AD FS，并将 Rancher 配置为使用 AD FS 服务器。如果需要获取在 Rancher 中设置 Microsoft AD FS 身份认证的指南，请参见： + +- [1. 在 Microsoft AD FS 中配置 Rancher](configure-ms-adfs-for-rancher.md) +- [2. 在 Rancher 中配置 Microsoft AD FS](configure-rancher-for-ms-adfs.md) + +:::note SAML 身份提供商注意事项 + +- SAML 协议不支持搜索或查找用户或组。因此，将用户或组添加到 Rancher 时不会对其进行验证。 +- 添加用户时，必须正确输入确切的用户 ID（即 `UID` 字段）。键入用户 ID 时，将不会搜索可能匹配的其他用户 ID。 +- 添加组时，必须从文本框旁边的下拉列表中选择组。Rancher 假定来自文本框的任何输入都是用户。 +- 用户组下拉列表仅显示你所属的用户组。如果你不是某个组的成员，你将无法添加该组。 + +::: + +### [后续操作：在 Microsoft AD FS 中配置 Rancher](configure-ms-adfs-for-rancher.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-microsoft-ad-federation-service-saml/configure-ms-adfs-for-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-microsoft-ad-federation-service-saml/configure-ms-adfs-for-rancher.md index 1bcf056acd8..83bcfee396b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-microsoft-ad-federation-service-saml/configure-ms-adfs-for-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-microsoft-ad-federation-service-saml/configure-ms-adfs-for-rancher.md @@ -2,7 +2,11 @@ title: 1. 在 Microsoft AD FS 中配置 Rancher --- -在配置 Rancher 以支持 AD FS 用户之前，你必须在 AD FS 中将 Rancher 添加为 [relying party trust](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/technical-reference/understanding-key-ad-fs-concepts)（信赖方信任）。 + + + + +在配置 Rancher 以支持 Active Directory Federation Service (AD FS) 之前，你必须在 AD FS 中将 Rancher 添加为 [relying party trust](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/technical-reference/understanding-key-ad-fs-concepts)（信赖方信任）。 1. 以管理用户身份登录 AD 服务器。 @@ -34,7 +38,7 @@ title: 1. 在 Microsoft AD FS 中配置 Rancher ![](/img/adfs/adfs-add-rpt-7.png) -1. 本教程不涉及多重身份验证。如果你想配置多重身份验证，请参见 [Microsoft 文档](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/configure-additional-authentication-methods-for-ad-fs)。 +1. 本教程不涉及多重身份认证。如果你想配置多重身份认证，请参见 [Microsoft 文档](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/configure-additional-authentication-methods-for-ad-fs)。 ![](/img/adfs/adfs-add-rpt-8.png) @@ -46,7 +50,6 @@ title: 1. 在 Microsoft AD FS 中配置 Rancher ![](/img/adfs/adfs-add-rpt-10.png) - 1. 选择 **Open the Edit Claim Rules...**。然后单击 **Close**。 ![](/img/adfs/adfs-add-rpt-11.png) @@ -61,17 +64,19 @@ title: 1. 在 Microsoft AD FS 中配置 Rancher 1. 将 **Claim rule name** 设置为所需的名称（例如 `Rancher Attributes`）并选择 **Active Directory** 作为 **Attribute store**。创建对应下表的映射： - | LDAP 属性 | 传出声明类型 | - | -------------------------------------------- | ------------------- | - | Given-Name | Given Name | - | User-Principal-Name | UPN | - | Token-Groups - Qualified by Long Domain Name | Group | - | SAM-Account-Name | 名称 | + | LDAP 属性 | 传出声明类型 | + | -------------------------------------------- | ------------ | + | Given-Name | Given Name | + | User-Principal-Name | UPN | + | Token-Groups - Qualified by Long Domain Name | Group | + | SAM-Account-Name | 名称 | +
![](/img/adfs/adfs-add-tcr-2.png) 1. 从 AD 服务器的以下位置下载 `federationmetadata.xml`： + ``` https:///federationmetadata/2007-06/federationmetadata.xml ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-microsoft-ad-federation-service-saml/configure-rancher-for-ms-adfs.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-microsoft-ad-federation-service-saml/configure-rancher-for-ms-adfs.md index b4a15d2bf4c..26e89e2058d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-microsoft-ad-federation-service-saml/configure-rancher-for-ms-adfs.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-microsoft-ad-federation-service-saml/configure-rancher-for-ms-adfs.md @@ -2,6 +2,10 @@ title: 2. 在 Rancher 中配置 Microsoft AD FS --- + + + + 完成[在 Microsoft AD FS 中配置 Rancher](configure-ms-adfs-for-rancher.md) 后，将你的 Active Directory Federation Service (AD FS) 信息输入 Rancher，以便 AD FS 用户可以通过 Rancher 进行身份认证。 :::note 配置 ADFS 服务器的重要说明： @@ -30,16 +34,15 @@ title: 2. 在 Rancher 中配置 Microsoft AD FS ## 配置 -| 字段 | 描述 | -|---------------------------|-----------------| -| 显示名称字段 | 包含用户显示名称的 AD 属性。

示例：`http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name` | -| 用户名字段 | 包含用户名/给定名称的 AD 属性。

示例：`http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname` | -| UID 字段 | 每个用户独有的 AD 属性。

示例：`http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn` | -| 用户组字段 | 创建用于管理组成员关系的条目。

示例：`http://schemas.xmlsoap.org/claims/Group` | -| Rancher API 主机 | Rancher Server 的 URL。 | -| 私钥/证书 | 在 Rancher 和你的 AD FS 之间创建安全外壳（SSH）的密钥/证书对。确保将 Common Name (CN) 设置为 Rancher Server URL。

[证书创建命令](#cert-command) | -| 元数据 XML | 从 AD FS 服务器导出的 `federationmetadata.xml` 文件。

你可以在 `https:///federationmetadata/2007-06/federationmetadata.xml` 找到该文件。 | - +| 字段 | 描述 | +| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| 显示名称字段 | 包含用户显示名称的 AD 属性。

示例：`http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name` | +| 用户名字段 | 包含用户名/给定名称的 AD 属性。

示例：`http://schemas.xmlsoap.org/ws/2005/05/identity/claims/givenname` | +| UID 字段 | 每个用户独有的 AD 属性。

示例：`http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn` | +| 用户组字段 | 创建用于管理组成员关系的条目。

示例：`http://schemas.xmlsoap.org/claims/Group` | +| Rancher API 主机 | Rancher Server 的 URL。 | +| 私钥/证书 | 在 Rancher 和你的 AD FS 之间创建安全外壳（SSH）的密钥/证书对。确保将 Common Name (CN) 设置为 Rancher Server URL。

[证书创建命令](#cert-command) | +| 元数据 XML | 从 AD FS 服务器导出的 `federationmetadata.xml` 文件。

你可以在 `https:///federationmetadata/2007-06/federationmetadata.xml` 找到该文件。 | @@ -51,4 +54,4 @@ title: 2. 在 Rancher 中配置 Microsoft AD FS openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com" ``` -::: \ No newline at end of file +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-openldap/configure-openldap.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-openldap/configure-openldap.md new file mode 100644 index 00000000000..7594371a296 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-openldap/configure-openldap.md @@ -0,0 +1,56 @@ +--- +title: 配置 OpenLDAP +--- + + + + + +如果你的组织使用 LDAP 进行认证，则可以配置 Rancher 与 OpenLDAP 服务器通信以对用户进行认证。这时 Rancher 管理员可以对外部用户系统中的用户和组进行集群和项目的访问控制，同时允许终端用户在登录 Rancher UI 时使用其 LDAP 凭据进行身份认证。 + +## 先决条件 + +必须为 Rancher 配置 LDAP 绑定账号（即 ServiceAccount），来搜索和检索应该具有访问权限的用户和组的 LDAP 条目。建议不要使用管理员账号或个人账号，而应在 OpenLDAP 中创建一个专用账号，该账号对配置的搜索库下的用户和组需要具有只读权限（参见下文）。 + +> **使用 TLS?** +> +> 如果 OpenLDAP 服务器使用的证书是自签名的或不是来自认可的证书颁发机构，请确保手头有 PEM 格式的 CA 证书（包含所有中间证书）。你必须在配置期间粘贴此证书，以便 Rancher 能够验证证书链。 + +## 在 Rancher 中配置 OpenLDAP + +配置 OpenLDAP 服务器，组和用户的设置。有关填写每个字段的帮助，请参见[配置参考](openldap-config-reference.md) + +> 在开始之前，请熟悉[外部认证配置和用户主体](../authentication-config/authentication-config.md#外部认证配置和用户主体)的概念。 + +1. 在左上角，单击 **☰ > 用户 & 认证**。 +1. 在左侧导航栏，单击**认证**。 +1. 单击 **OpenLDAP**。填写**配置 OpenLDAP 服务器**表单。 +1. 点击**启用**。 + +### 测试认证 + +完成配置后，请测试与 OpenLDAP 服务器的连接。如果测试成功，则表明 OpenLDAP 认证已启用。 + +:::note + +于此步骤中输入的 OpenLDAP 用户凭证将映射到本地主体账号，并在 Rancher 中分配系统管理员权限。因此，你应该决定使用哪个 OpenLDAP 账号来执行此步骤。 + +::: + +1. 输入应映射到本地主体账号的 OpenLDAP 账号的**用户名**和**密码** 。 +2. 点击**启用 OpenLDAP 认证**来测试 OpenLDAP 的连接并完成设置。 + +**结果**： + +- OpenLDAP 认证配置成功。 +- 与输入凭证对应的 LDAP 用户被映射到本地主体（管理员）账号。 + +:::note + +如果 LDAP 服务中断，你仍然可以使用本地配置的 `admin` 账号和密码登录。 + +::: + +## 附录：故障排除 + +如果在测试与 OpenLDAP 服务器的连接时遇到问题，请首先仔细检查为 ServiceAccount 输入的凭证以及搜索库配置。你还可以检查 Rancher 日志来查明问题的原因。调试日志可能包含有关错误的更详细信息。详情请参见[如何启用调试日志](../../../../faq/technical-items.md#how-can-i-enable-debug-logging)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-openldap/openldap-config-reference.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-openldap/openldap-config-reference.md index 581a85efc2c..24c338fe708 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-openldap/openldap-config-reference.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-openldap/openldap-config-reference.md @@ -2,17 +2,20 @@ title: OpenLDAP 配置参考 --- + + + + 有关配置 OpenLDAP 认证的更多详细信息，请参见[官方文档](https://www.openldap.org/doc/)。 -> 在开始之前，请熟悉[外部身份验证配置和主体用户](../../../../pages-for-subheaders/authentication-config.md#外部身份验证配置和用户主体)的概念。 - +> 在开始之前，请熟悉[外部认证配置和用户主体](../authentication-config/authentication-config.md#外部认证配置和用户主体)的概念。 ## 背景：OpenLDAP 认证流程 1. 当用户尝试使用其 LDAP 凭证登录时，Rancher 会使用具有搜索目录和读取用户/组属性权限的 ServiceAccount，创建与 LDAP 服务器的初始绑定。 2. 然后，Rancher 使用搜索筛选器根据用户名和配置的属性映射为用户搜索目录。 -3. 找到用户后，将使用用户的 DN 和提供的密码，通过另一个 LDAP 绑定请求对用户进行身份验证。 -4. 身份验证成功后，Rancher 将基于用户对象的成员属性和配置的用户映射属性执行组搜索，来解析组成员。 +3. 找到用户后，将使用用户的 DN 和提供的密码，通过另一个 LDAP 绑定请求对用户进行身份认证。 +4. 认证成功后，Rancher 将基于用户对象的成员属性和配置的用户映射属性执行组搜索，来解析组成员。 ## OpenLDAP 服务器配置 @@ -26,16 +29,16 @@ title: OpenLDAP 配置参考
OpenLDAP 服务器参数
-| 参数 | 描述 | -|:--|:--| -| 主机名 | 指定 OpenLDAP 服务器的主机名或 IP 地址。 | -| 端口 | 指定 OpenLDAP 服务器监听连接的端口。未加密的 LDAP 通常使用 389 的标准端口，而 LDAPS 使用 636 端口。 | -| TLS | 选中此框可启用 SSL/TLS 上的 LDAP（通常称为 LDAPS）。如果服务器使用自签名/企业签名的证书，则还需要粘贴 CA 证书。 | -| 服务器连接超时 | Rancher 在认为无法访问服务器之前等待的时间（秒）。 | -| ServiceAccount 标识名称 | 输入用于绑定，搜索和检索 LDAP 条目的用户的标识名称（DN）。 | -| ServiceAccount 密码 | ServiceAccount 的密码。 | -| 用户搜索库 | 输入目录树中开始搜索用户对象的节点的标识名称（DN）。所有用户都必须是此基础标识名称的后代。例如，"ou=people,dc=acme,dc=com"。 | -| 组搜索库 | 如果组位于`用户搜索库`下配置的节点之外的其他节点下，则需要在此处提供标识名称。否则，将此字段留空。例如："ou=groups,dc=acme,dc=com"。 | +| 参数 | 描述 | +| :---------------------- | :----------------------------------------------------------------------------------------------------------------------------------- | +| 主机名 | 指定 OpenLDAP 服务器的主机名或 IP 地址。 | +| 端口 | 指定 OpenLDAP 服务器监听连接的端口。未加密的 LDAP 通常使用 389 的标准端口，而 LDAPS 使用 636 端口。 | +| TLS | 选中此框可启用 SSL/TLS 上的 LDAP（通常称为 LDAPS）。如果服务器使用自签名/企业签名的证书，则还需要粘贴 CA 证书。 | +| 服务器连接超时 | Rancher 在认为无法访问服务器之前等待的时间（秒）。 | +| ServiceAccount 标识名称 | 输入用于绑定，搜索和检索 LDAP 条目的用户的标识名称（DN）。 | +| ServiceAccount 密码 | ServiceAccount 的密码。 | +| 用户搜索库 | 输入目录树中开始搜索用户对象的节点的标识名称（DN）。所有用户都必须是此基础标识名称的后代。例如，"ou=people,dc=acme,dc=com"。 | +| 组搜索库 | 如果组位于`用户搜索库`下配置的节点之外的其他节点下，则需要在此处提供标识名称。否则，将此字段留空。例如："ou=groups,dc=acme,dc=com"。 | ## 用户/组 Schema 配置 @@ -51,15 +54,15 @@ title: OpenLDAP 配置参考
用户 Schema 配置参数
-| 参数 | 描述 | -|:--|:--| -| Object Class | 域中用于用户对象的对象类别名称。如果定义了此参数，则仅指定对象类别的名称 - *请勿*将其放在 LDAP 包装器中，例如 `&(objectClass=xxxx)`。 | -| Username Attribute | 用户属性的值适合作为显示名称。 | -| Login Attribute | 登录属性的值与用户登录 Rancher 时输入的凭证的用户名部分匹配。通常是 `uid`。 | -| User Member Attribute | 包含用户所属组的标识名称的用户属性。通常是 `memberOf` 或 `isMemberOf`。 | -| Search Attribute | 当用户输入文本以在用户界面中添加用户或组时，Rancher 会查询 LDAP 服务器，并尝试根据此设置中提供的属性匹配用户。可以通过使用管道（“\|”）符号分隔属性来指定多个属性。 | -| User Enabled Attribute | 如果 OpenLDAP 服务器的 Schema 支持使用用户属性的值来评估账号是禁用还是关闭，请输入该属性的名称。默认的 OpenLDAP Schema 不支持此功能，因此此字段通常留空。 | -| Disabled Status Bitmask | 禁用/锁定的用户账号的值。如果 `User Enabled Attribute` 是空的，则忽略此参数。 | +| 参数 | 描述 | +| :---------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Object Class | 域中用于用户对象的对象类别名称。如果定义了此参数，则仅指定对象类别的名称 - *请勿*将其放在 LDAP 包装器中，例如 `&(objectClass=xxxx)`。 | +| Username Attribute | 用户属性的值适合作为显示名称。 | +| Login Attribute | 登录属性的值与用户登录 Rancher 时输入的凭证的用户名部分匹配。通常是 `uid`。 | +| User Member Attribute | 包含用户所属组的标识名称的用户属性。通常是 `memberOf` 或 `isMemberOf`。 | +| Search Attribute | 当用户输入文本以在用户界面中添加用户或组时，Rancher 会查询 LDAP 服务器，并尝试根据此设置中提供的属性匹配用户。可以通过使用管道（“\|”）符号分隔属性来指定多个属性。 | +| User Enabled Attribute | 如果 OpenLDAP 服务器的 Schema 支持使用用户属性的值来评估账号是禁用还是关闭，请输入该属性的名称。默认的 OpenLDAP Schema 不支持此功能，因此此字段通常留空。 | +| Disabled Status Bitmask | 禁用/锁定的用户账号的值。如果 `User Enabled Attribute` 是空的，则忽略此参数。 | ### 组 Schema 配置 @@ -67,12 +70,12 @@ title: OpenLDAP 配置参考
组 Schema 配置参数
-| 参数 | 描述 | -|:--|:--| -| Object Class | 域中用于组条目的对象类别名称。如果定义了此参数，则仅指定对象类别的名称 - *请勿*将其放在 LDAP 包装器中，例如 `&(objectClass=xxxx)`。 | -| Name Attribute | 名称属性的值适合作为显示名称。 | -| Group Member User Attribute | **用户属性**的名称。它的格式与 `Group Member Mapping Attribute` 中的组成员匹配。 | -| Group Member Mapping Attribute | 包含组成员的组属性的名称。 | -| Search Attribute | 在 UI 中将组添加到集群或项目时，用于构造搜索筛选器的属性。请参见用户 Schema 的 `Search Attribute`。 | -| Group DN Attribute | 组属性的名称，其格式与用户的组成员属性中的值匹配。参见 `User Member Attribute`。 | -| Nested Group Membership | 此设置定义 Rancher 是否应解析嵌套组成员身份。仅当你的组织使用这些嵌套成员身份时才使用（即你有包含其他组作为成员的组）。如果你使用 Shibboleth，此选项会被禁用。 | \ No newline at end of file +| 参数 | 描述 | +| :----------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Object Class | 域中用于组条目的对象类别名称。如果定义了此参数，则仅指定对象类别的名称 - *请勿*将其放在 LDAP 包装器中，例如 `&(objectClass=xxxx)`。 | +| Name Attribute | 名称属性的值适合作为显示名称。 | +| Group Member User Attribute | **用户属性**的名称。它的格式与 `Group Member Mapping Attribute` 中的组成员匹配。 | +| Group Member Mapping Attribute | 包含组成员的组属性的名称。 | +| Search Attribute | 在 UI 中将组添加到集群或项目时，用于构造搜索筛选器的属性。请参见用户 Schema 的 `Search Attribute`。 | +| Group DN Attribute | 组属性的名称，其格式与用户的组成员属性中的值匹配。参见 `User Member Attribute`。 | +| Nested Group Membership | 此设置定义 Rancher 是否应解析嵌套组成员身份。仅当你的组织使用这些嵌套成员身份时才使用（即你有包含其他组作为成员的组）。如果你使用 Shibboleth，此选项会被禁用。 | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-shibboleth-saml/about-group-permissions.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-shibboleth-saml/about-group-permissions.md index 69155120d28..4321f1e6765 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-shibboleth-saml/about-group-permissions.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-shibboleth-saml/about-group-permissions.md @@ -2,14 +2,18 @@ title: Shibboleth 和 OpenLDAP 的组权限 --- + + + + 由于 Shibboleth 是 SAML 提供者，因此它不支持搜索用户组的功能。虽然 Shibboleth 集成可以验证用户凭证，但是如果没有其他配置，Shibboleth 不能在 Rancher 中给用户组分配权限。 -你可以通过配置 OpenLDAP 标识提供者来解决这个问题。如果让 Shibboleth 使用 OpenLDAP 后端，你将能够在 Rancher 中搜索组，并从 Rancher UI 将集群、项目或命名空间等资源分配给用户组。 +你可以通过配置 OpenLDAP 来解决这个问题。如果让 Shibboleth 使用 OpenLDAP 后端，你将能够在 Rancher 中搜索组，并从 Rancher UI 将集群、项目或命名空间等资源分配给用户组。 ### 名词解释 - **Shibboleth**：用于计算机网络和互联网的单点登录系统。它允许用户仅使用一种身份登录到各种系统。它验证用户凭证，但不单独处理组成员身份。 -- **SAML**：安全声明标记语言（Security Assertion Markup Language），用于在身份提供程序和服务提供商之间交换身份验证和授权数据的开放标准。 +- **SAML**：安全声明标记语言（Security Assertion Markup Language），用于在身份提供程序和服务提供商之间交换认证和授权数据的开放标准。 - **OpenLDAP**：轻型目录访问协议（LDAP）的免费开源实现。它用于管理组织的计算机和用户。OpenLDAP 对 Rancher 用户很有用，因为它支持组。只要组已存在于身份提供程序中，你就可以在 Rancher 中为组分配权限，从而让组访问资源（例如集群，项目或命名空间）。 - **IdP 或 IDP**：身份提供程序。OpenLDAP 是身份提供程序的一个例子。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-shibboleth-saml/configure-shibboleth-saml.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-shibboleth-saml/configure-shibboleth-saml.md new file mode 100644 index 00000000000..285a5d3e6aa --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/configure-shibboleth-saml/configure-shibboleth-saml.md @@ -0,0 +1,104 @@ +--- +title: 配置 Shibboleth (SAML) +--- + + + + + +如果你的组织使用 Shibboleth Identity Provider (IdP) 进行用户身份认证，你可以通过配置 Rancher 来允许用户使用 Shibboleth 凭证登录。 + +在此配置中，当 Rancher 用户登录时，他们将被重定向到 Shibboleth IdP 来输入凭证。认证结束后，他们将被重定向回 Rancher UI。 + +如果你将 OpenLDAP 配置为 Shibboleth 的后端，SAML 断言会返回到 Rancher，其中包括用于引用组的用户属性。然后，通过认证的用户将能够访问其所在的组有权访问的 Rancher 资源。 + +> 本节假定你已了解 Rancher、Shibboleth 和 OpenLDAP 是如何协同工作的。有关工作原理的详细说明，请参见[本页](about-group-permissions.md) + +# 在 Rancher 中设置 Shibboleth + +### Shibboleth 先决条件 + +> - 你必须配置了 Shibboleth IdP 服务器。 +> - 以下是 Rancher Service Provider 配置所需的 URL： +> 元数据 URL：`https:///v1-saml/shibboleth/saml/metadata` +> 断言使用者服务 (ACS) URL：`https:///v1-saml/shibboleth/saml/acs` +> - 从 IdP 服务器导出 `metadata.xml` 文件。详情请参见 [Shibboleth 文档](https://wiki.shibboleth.net/confluence/display/SP3/Home)。 + +### 在 Rancher 中配置 Shibboleth + +如果你的组织使用 Shibboleth 进行用户身份认证，你可以通过配置 Rancher 来允许你的用户使用 IdP 凭证登录。 + +1. 在左上角，单击 **☰ > 用户 & 认证**。 +1. 在左侧导航栏，单击**认证**。 +1. 单击 **Shibboleth**。 +1. 填写**配置 Shibboleth 账号**表单。Shibboleth IdP 允许你指定要使用的数据存储。你可以添加数据库或使用现有的 ldap 服务器。例如，如果你选择 Active Directory (AD) 服务器，下面的示例将描述如何将 AD 属性映射到 Rancher 中的字段： + + 1. **显示名称字段**：包含用户显示名称的 AD 属性（例如：`displayName`）。 + + 1. **用户名字段**：包含用户名/给定名称的 AD 属性（例如：`givenName`）。 + + 1. **UID 字段**：每个用户唯一的 AD 属性（例如：`sAMAccountName`、`distinguishedName`）。 + + 1. **用户组字段**: 创建用于管理组成员关系的条目（例如：`memberOf`）。 + + 1. **Rancher API 主机**：你的 Rancher Server 的 URL。 + + 1. **私钥**和**证书**：密钥/证书对，用于在 Rancher 和你的 IdP 之间创建一个安全外壳（SSH）。 + + 你可以使用 openssl 命令进行创建。例如： + + ``` + openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com" + ``` + + 1. **IDP 元数据**：从 IdP 服务器导出的 `metadata.xml` 文件。 + +1. 完成**配置 Shibboleth 账号**表单后，单击**启用**。 + + Rancher 会将你重定向到 IdP 登录页面。输入使用 Shibboleth IdP 的用户凭证，来验证你的 Rancher Shibboleth 配置。 + + :::note + + 你可能需要禁用弹出窗口阻止程序才能看到 IdP 登录页面。 + + ::: + +**结果**：已将 Rancher 配置为使用 Shibboleth。你的用户现在可以使用 Shibboleth 登录名登录 Rancher。 + +### SAML 提供商注意事项 + +SAML 协议不支持用户或用户组的搜索或查找。因此，如果你没有为 Shibboleth 配置 OpenLDAP，则请留意以下警告。 + +- 在 Rancher 中为用户或组分配权限时，不会对用户或组进行验证。 +- 添加用户时，必须正确输入准确的用户 ID（即 UID 字段）。在你输入用户 ID 时，将不会搜索可能匹配的其他用户 ID。 +- 添加组时，必须从文本框旁边的下拉列表中选择组。Rancher 假定来自文本框的任何输入都是用户。 +- 用户组下拉列表仅显示你所属的用户组。如果你不是某个组的成员，你将无法添加该组。 + +要在 Rancher 中分配权限时启用搜索组，你需要为 SAML 身份认证服务配置支持组的后端（例如 OpenLDAP）。 + +# 在 Rancher 中设置 OpenLDAP + +如果你将 OpenLDAP 配置为 Shibboleth 的后端，SAML 断言会返回到 Rancher，其中包括用于引用组的用户属性。然后，通过认证的用户将能够访问其所在的组有权访问的 Rancher 资源。 + +### OpenLDAP 先决条件 + +必须为 Rancher 配置 LDAP 绑定账号（即 ServiceAccount），来搜索和检索应该具有访问权限的用户和组的 LDAP 条目。建议不要使用管理员账号或个人账号，而应在 OpenLDAP 中创建一个专用账号，该账号对配置的搜索库下的用户和组需要具有只读权限（参见下文）。 + +> **使用 TLS？** +> +> 如果 OpenLDAP 服务器使用的证书是自签名的或不是来自认可的证书颁发机构，请确保手头有 PEM 格式的 CA 证书（包含所有中间证书）。你必须在配置期间粘贴此证书，以便 Rancher 能够验证证书链。 + +### 在 Rancher 中配置 OpenLDAP + +配置 OpenLDAP 服务器，组和用户的设置。有关填写每个字段的帮助，请参见[配置参考](../configure-openldap/openldap-config-reference.md)。请注意，嵌套组成员资格不适用于 Shibboleth。 + +> 在开始之前，请熟悉[外部认证配置和用户主体](../authentication-config/authentication-config.md#外部认证配置和用户主体)的概念。 + +1. 使用初始的本地 `admin` 账号登录到 Rancher UI。 +1. 在左上角，单击 **☰ > 用户 & 认证**。 +1. 在左侧导航栏，单击**认证**。 +1. 单击 **OpenLDAP**。将显示**配置 OpenLDAP 服务器**表单。 + +## 故障排除 + +如果在测试与 OpenLDAP 服务器的连接时遇到问题，请首先仔细检查为 ServiceAccount 输入的凭证以及搜索库配置。你还可以检查 Rancher 日志来查明问题的原因。调试日志可能包含有关错误的更详细信息。详情请参见[如何启用调试日志](../../../../faq/technical-items.md#how-can-i-enable-debug-logging)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md index 3fb65c5d7e0..e0f82294b8d 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md @@ -2,6 +2,10 @@ title: 全局权限 --- + + + + _权限_ 是你在为用户选择自定义权限时可以分配的个人访问权限。 全局权限定义用户在任何特定集群之外的授权。Rancher 提供四种开箱即用的默认全局权限：`Administrator` (管理员)、`Restricted Admin` (受限管理员)、`Standard User` (标准用户) 和 `User-Base` 用户。 @@ -29,55 +33,55 @@ Rancher 还增加了一个新设置，来将初始启动的管理员设置为 `r ``` CATTLE_RESTRICTED_DEFAULT_ADMIN=true ``` + ### `restricted-admin` 的权限列表 下表列出了 `restricted-admin` 与 `Administrator` 和 `Standard User` 角色相比应具有的权限和操作： -| 类别 | 操作 | 全局管理员 | 普通用户 | 受限管理员 | 受限管理员的注意事项 | -| -------- | ------ | ------------ | ------------- | ---------------- | ------------------------------- | -| 本地集群功能 | 管理本地集群（列出、编辑、导入主机） | 是 | 否 | 否 | | -| | 创建项目/命名空间 | 是 | 否 | 否 | | -| | 添加集群/项目成员 | 是 | 否 | 否 | | -| | 全局 DNS | 是 | 否 | 否 | | -| | 访问 CRD 和 CR 的管理集群 | 是 | 否 | 是 | | -| | 另存为 RKE 模板 | 是 | 否 | 否 | | -| 安全 | | | | | | -| 启用身份验证 | 配置身份验证 | 是 | 否 | 是 | | -| 角色 | 创建/分配 GlobalRoles | 是 | 否（可列出） | 是 | 认证 Webhook 允许为已经存在的权限创建 globalrole | -| | 创建/分配 ClusterRoles | 是 | 否（可列出） | 是 | 不在本地集群中 | -| | 创建/分配 ProjectRoles | 是 | 否（可列出） | 是 | 不在本地集群中 | -| 用户 | 添加用户/编辑/删除/停用用户 | 是 | 否 | 是 | | -| 组 | 将全局角色分配给组 | 是 | 否 | 是 | 在 Webhook 允许的范围内 | -| | 刷新组 | 是 | 否 | 是 | | -| PSP | 管理 PSP 模板 | 是 | 否（可列出） | 是 | 与 PSP 的全局管理员权限相同 | -| 工具 | | | | | | -| | 管理 RKE 模板 | 是 | 否 | 是 | | -| | 管理全局应用商店 | 是 | 否 | 是 | 无法编辑/删除内置系统应用商店。可以管理 Helm 库 | -| | 集群驱动 | 是 | 否 | 是 | | -| | 主机驱动 | 是 | 否 | 是 | | -| | GlobalDNS 提供商 | 是 | 是（自己） | 是 | | -| | GlobalDNS 条目 | 是 | 是（自己） | 是 | | -| 设置 | | | | | | -| | 管理设置 | 是 | 否（可列出） | 否（可列出） | | -| 用户 | | | | | | -| | 管理 API 密钥 | 是（管理所有） | 是（管理自己的） | 是（管理自己的） | | -| | 管理节点模板 | 是 | 是（管理自己的） | 是（管理自己的） | 只能管理自己的节点模板，不能管理其他用户创建的节点模板。 | -| | 管理云凭证 | 是 | 是（管理自己的） | 是（管理自己的） | 只能管理自己的云凭证，不能管理其他用户创建的云凭证。 | -| 下游集群 | 创建集群 | 是 | 是 | 是 | | -| | 编辑集群 | 是 | 是 | 是 | | -| | 轮换证书 | 是 | | 是 | | -| | 立即创建快照 | 是 | | 是 | | -| | 恢复快照 | 是 | | 是 | | -| | 另存为 RKE 模板 | 是 | 否 | 是 | | -| | 运行 CIS 扫描 | 是 | 是 | 是 | | -| | 添加成员 | 是 | 是 | 是 | | -| | 创建项目 | 是 | 是 | 是 | | -| 自 2.5 起的功能 Chart | | | | | | -| | 安装 Fleet | 是 | | 是 | 无法在本地集群中运行 Fleet | -| | 部署 EKS 集群 | 是 | 是 | 是 | | -| | 部署 GKE 集群 | 是 | 是 | 是 | | -| | 部署 AKS 集群 | 是 | 是 | 是 | | - +| 类别 | 操作 | 全局管理员 | 普通用户 | 受限管理员 | 受限管理员的注意事项 | +| --------------------- | ------------------------------------ | -------------- | ---------------- | ---------------- | -------------------------------------------------------- | +| 本地集群功能 | 管理本地集群（列出、编辑、导入主机） | 是 | 否 | 否 | | +| | 创建项目/命名空间 | 是 | 否 | 否 | | +| | 添加集群/项目成员 | 是 | 否 | 否 | | +| | 全局 DNS | 是 | 否 | 否 | | +| | 访问 CRD 和 CR 的管理集群 | 是 | 否 | 是 | | +| | 另存为 RKE 模板 | 是 | 否 | 否 | | +| 安全 | | | | | | +| 启用身份验证 | 配置身份验证 | 是 | 否 | 是 | | +| 角色 | 创建/分配 GlobalRoles | 是 | 否（可列出） | 是 | 认证 Webhook 允许为已经存在的权限创建 globalrole | +| | 创建/分配 ClusterRoles | 是 | 否（可列出） | 是 | 不在本地集群中 | +| | 创建/分配 ProjectRoles | 是 | 否（可列出） | 是 | 不在本地集群中 | +| 用户 | 添加用户/编辑/删除/停用用户 | 是 | 否 | 是 | | +| 组 | 将全局角色分配给组 | 是 | 否 | 是 | 在 Webhook 允许的范围内 | +| | 刷新组 | 是 | 否 | 是 | | +| PSP | 管理 PSP 模板 | 是 | 否（可列出） | 是 | 与 PSP 的全局管理员权限相同 | +| 工具 | | | | | | +| | 管理 RKE 模板 | 是 | 否 | 是 | | +| | 管理全局应用商店 | 是 | 否 | 是 | 无法编辑/删除内置系统应用商店。可以管理 Helm 库 | +| | 集群驱动 | 是 | 否 | 是 | | +| | 主机驱动 | 是 | 否 | 是 | | +| | GlobalDNS 提供商 | 是 | 是（自己） | 是 | | +| | GlobalDNS 条目 | 是 | 是（自己） | 是 | | +| 设置 | | | | | | +| | 管理设置 | 是 | 否（可列出） | 否（可列出） | | +| 用户 | | | | | | +| | 管理 API 密钥 | 是（管理所有） | 是（管理自己的） | 是（管理自己的） | | +| | 管理节点模板 | 是 | 是（管理自己的） | 是（管理自己的） | 只能管理自己的节点模板，不能管理其他用户创建的节点模板。 | +| | 管理云凭证 | 是 | 是（管理自己的） | 是（管理自己的） | 只能管理自己的云凭证，不能管理其他用户创建的云凭证。 | +| 下游集群 | 创建集群 | 是 | 是 | 是 | | +| | 编辑集群 | 是 | 是 | 是 | | +| | 轮换证书 | 是 | | 是 | | +| | 立即创建快照 | 是 | | 是 | | +| | 恢复快照 | 是 | | 是 | | +| | 另存为 RKE 模板 | 是 | 否 | 是 | | +| | 运行 CIS 扫描 | 是 | 是 | 是 | | +| | 添加成员 | 是 | 是 | 是 | | +| | 创建项目 | 是 | 是 | 是 | | +| 自 2.5 起的功能 Chart | | | | | | +| | 安装 Fleet | 是 | | 是 | 无法在本地集群中运行 Fleet | +| | 部署 EKS 集群 | 是 | 是 | 是 | | +| | 部署 GKE 集群 | 是 | 是 | 是 | | +| | 部署 AKS 集群 | 是 | 是 | 是 | | ### 将全局管理员更改为受限管理员 @@ -89,7 +93,7 @@ CATTLE_RESTRICTED_DEFAULT_ADMIN=true ## 分配全局权限 -本地用户的全局权限分配与使用外部身份验证登录 Rancher 的用户不同。 +本地用户的全局权限分配与使用外部认证登录 Rancher 的用户不同。 ### 新本地用户的全局权限 @@ -101,11 +105,11 @@ CATTLE_RESTRICTED_DEFAULT_ADMIN=true 1. 在左侧导航栏中，单击**角色**。 1. **角色**页面有按范围分组的角色选项卡。每个表都列出了范围内的角色。在**全局**选项卡的**新用户的默认角色**列中，默认授予新用户的权限用复选标记表示。 -你可以[更改默认全局权限来满足你的需要](#配置默认全局权限)。 +你可以[更改默认全局权限来满足你的需要](#配置默认的全局权限) -### 使用外部身份验证程序的用户的全局权限 +### 使用外部认证登录的用户的全局权限 -当用户首次使用外部身份验证提供程序登录 Rancher 时，他们会自动分配到**新用户的默认角色**的全局权限。默认情况下，Rancher 为新用户分配 **Standard User** 权限。 +当用户首次使用外部认证登录 Rancher 时，他们会自动分配到**新用户的默认角色**的全局权限。默认情况下，Rancher 为新用户分配 **Standard User** 权限。 如果需要查看新用户的默认权限： @@ -113,27 +117,27 @@ CATTLE_RESTRICTED_DEFAULT_ADMIN=true 1. 在左侧导航栏中，单击**角色**。 1. **角色**页面有按范围分组的角色选项卡。每个表都列出了范围内的角色。在每个页面的**新用户的默认角色**列中，默认授予新用户的权限用复选标记表示。 -你可以[更改默认权限来满足你的需要](#配置默认全局权限)。 +你可以[更改默认权限来满足你的需要](#配置默认的全局权限) -权限可以[分配](#为单个用户配置全局权限)给单个用户。 +你可以按照[步骤](#为单个用户配置全局权限)操作来将权限分配给单个用户。 -如果外部身份验证提供程序支持组，你可以[同时为组中的每个成员分配角色](#为组配置全局权限)。 +如果外部认证服务支持组，你可以[同时为组中的每个成员分配角色](#为组配置全局权限)。 ## 自定义全局权限 -使用自定义权限可以为用户配置在 Rancher 中的更受限或指定的访问权限。 +使用自定义权限可以为用户提供 Rancher 中更为受限或特定的访问权限。 -当来自[外部身份验证系统](../../../../pages-for-subheaders/authentication-config.md)的用户首次登录 Rancher 时，他们会自动分配到一组全局权限（以下简称权限）。默认情况下，用户第一次登录后会被创建为用户，并分配到默认的`用户`权限。标准的`用户`权限允许用户登录和创建集群。 +当来自[外部认证](../authentication-config/authentication-config.md)的用户首次登录 Rancher 时，他们会自动分配到一组全局权限（以下简称权限）。默认情况下，用户第一次登录后会被创建为用户，并分配到默认的`用户`权限。标准的`用户`权限允许用户登录和创建集群。 但是，在某些组织中，这些权限可能会被认为权限过大。你可以为用户分配一组更具限制性的自定义全局权限，而不是为用户分配 `Administrator` 或 `Standard User` 的默认全局权限。 -默认角色（管理员和标准用户）都内置了多个全局权限。系统管理员角色包括所有全局权限，而默认用户角色包括三个全局权限，分别是创建集群，使用应用商店模板，和 User Base（登录 Rancher 的最低权限）。换句话说，自定义全局权限是模块化的，因此，如果你要更改默认用户角色权限，你可以选择需要包括在新的默认用户角色中的全局权限子集。 +默认角色（管理员和标准用户）都内置了多个全局权限。系统管理员角色包括所有全局权限，而默认用户角色包括三个全局权限，分别是创建集群、使用应用商店模板和 User Base（登录 Rancher 的最低权限）。换句话说，自定义全局权限是模块化的，因此，如果你要更改默认用户角色权限，你可以选择需要包括在新的默认用户角色中的全局权限子集。 管理员可以通过多种方式强制执行自定义全局权限： -- [更改新用户的默认权限](#配置默认全局权限) -- [为单个用户配置全局权限](#为单个用户配置全局权限) -- [为组配置全局权限](#为组配置全局权限) +- [更改新用户的默认权限](#配置默认的全局权限). +- [为单个用户配置全局权限](#为单个用户配置全局权限). +- [为组配置全局权限](#为组配置全局权限). ### 自定义全局权限参考 @@ -173,7 +177,7 @@ CATTLE_RESTRICTED_DEFAULT_ADMIN=true :::note -默认角色仅分配给从外部身份验证提供程序添加的用户。对于本地用户，在将用户添加到 Rancher 时，必须显式分配全局权限。你可以在添加用户时自定义这些全局权限。 +默认角色仅分配给从外部认证登录的用户。对于本地用户，在将用户添加到 Rancher 时，必须显式分配全局权限。你可以在添加用户时自定义这些全局权限。 ::: @@ -206,17 +210,17 @@ CATTLE_RESTRICTED_DEFAULT_ADMIN=true 对于现有用户，新权限将在用户退出 Rancher 并重新登录时，或当管理员[刷新用户组成员名单](#刷新用户组成员名单)时生效。 -对于新用户，新权限在用户首次登录 Rancher 时生效。除了**新用户的默认角色**全局权限外，来自该组的新用户还将获得自定义全局角色的权限。默认情况下，**新用户的默认角色**权限等同于 **Standard User** 全局角色，但默认权限可以[配置。](#配置默认全局权限) +对于新用户，新权限在用户首次登录 Rancher 时生效。除了**新用户的默认角色**全局权限外，来自该组的新用户还将获得自定义全局角色的权限。默认情况下，**新用户的默认角色**权限等同于 **Standard User** 全局角色，但默认权限可以[配置。](#配置默认的全局权限) -如果从外部身份验证提供程序组中删除用户，该用户将失去分配给该组的自定义全局角色的权限。他们将继续拥有分配给他们的所有剩余角色，这通常包括标记为**新用户的默认角色**的角色。Rancher 将在用户登出或管理员[刷新用户组成员名单](#刷新用户组成员名单)时删除与组关联的权限。 +如果从外部认证服务中将用户从组中删除，该用户将失去分配给该组的自定义全局角色的权限。他们将继续拥有分配给他们的其他剩余角色，这通常包括标记为**新用户的默认角色**的角色。Rancher 将在用户登出或管理员[刷新用户组成员名单](#刷新用户组成员名单)时删除与组关联的权限。 :::note 先决条件： 只有在以下情况下，你才能将全局角色分配给组： -* 你已设置[外部身份验证提供程序](../../../../pages-for-subheaders/authentication-config.md#外部验证与本地验证)。 -* 外部身份验证提供程序支持[用户组](../authentication-config/manage-users-and-groups.md)。 -* 你已使用身份验证提供程序设置了至少一个用户组。 +- 你已设置[外部认证](../authentication-config/authentication-config.md#external-vs-local-authentication) +- 外部认证服务支持[用户组](../authentication-config/manage-users-and-groups.md) +- 你已使用外部认证服务设置了至少一个用户组。 ::: @@ -236,7 +240,7 @@ CATTLE_RESTRICTED_DEFAULT_ADMIN=true 如果要让更改立即生效，管理员或集群所有者可以刷新用户组成员名单。 -如果用户已经从外部身份验证服务中的组中删除，管理员也可能需要刷新用户组成员名单。在这种情况下，刷新操作会让 Rancher 知道用户已从组中删除。 +如果用户已经从外部认证服务中的组中删除，管理员也需要刷新用户组成员名单。在这种情况下，刷新操作会让 Rancher 知道用户已从组中删除。 要刷新用户组成员名单： diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md new file mode 100644 index 00000000000..0e8a5e89824 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md @@ -0,0 +1,29 @@ +--- +title: 管理 RBAC +--- + + + + + +Rancher 通过 _用户_ 进行授权管理。如[认证](../authentication-config/authentication-config.md)中所述，用户可以是本地用户，也可以是外部用户。 + +配置外部认证后，**用户**页面上显示的用户会发生变化。 + +- 如果你以本地用户身份登录，则仅显示本地用户。 + +- 如果你以外部用户身份登录，则会同时显示外部用户和本地用户。 + +## 用户和角色 + +一旦用户登录到 Rancher，他们的 _授权_，也就是他们在系统中的访问权限，将由 _全局权限_ 和 _集群和项目角色_ 决定。 + +- [全局权限](global-permissions.md): + + 定义用户在任何特定集群之外的授权。 + +- [集群和项目角色](cluster-and-project-roles.md): + + 定义用户在分配了角色的特定集群或项目中的授权。 + +全局权限以及集群和项目角色都是基于 [Kubernetes RBAC](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) 实现的。因此，权限和角色的底层实现是由 Kubernetes 完成的。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher-launched-kubernetes-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher-launched-kubernetes-clusters.md index 11b0dc2a9fb..3ba17f6ed64 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher-launched-kubernetes-clusters.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher-launched-kubernetes-clusters.md @@ -256,11 +256,19 @@ Rancher 支持两种不同的备份目标： ### S3 备份目标 -`S3` 备份目标允许用户配置与 S3 兼容的后端来存储快照。此选项的主要好处是，如果集群丢失了所有 etcd 节点，由于快照存储在外部，集群仍然可以恢复。Rancher 推荐使用 `S3` 备份这类外部目标。但是它的配置要求进行额外的操作，这也应该被考虑在其中。由于 Rancher 会为群集配置的 S3 存储桶/文件夹中列出的任何可用快照填充快照信息，因此建议你确保每个集群都具有唯一的存储桶或文件夹。 +我们建议你使用 `S3` 备份目标。你可以将快照存储在外部 S3 兼容的后端上。由于快照不存储在本地，因此即使丢失所有 etcd 节点，你仍然可以还原集群。 + +虽然 `S3` 比本地备份具有优势，但它需要额外的配置。 + +:::caution + +如果你使用 S3 备份目标，请确保每个集群都有自己的存储桶或文件夹。Rancher 将使用集群配置的 S3 存储桶或文件夹中的可用快照来填充快照信息。 + +::: | 选项 | 描述 | 必填 | |---|---|---| -| S3 存储桶名称 | 将存储备份的 S3 存储桶名称 | * | +| S3 存储桶名称 | 用于存储备份的 S3 存储桶名称 | * | | S3 区域 | 备份存储桶的 S3 区域 | | | S3 区域端点 | 备份存储桶的 S3 区域端点 | * | | S3 访问密钥 | 有权访问备份存储桶的 S3 访问密钥 | * | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md new file mode 100644 index 00000000000..d99c8f91f92 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md @@ -0,0 +1,100 @@ +--- +title: 备份和灾难恢复 +keywords: [rancher 备份还原, rancher 备份与恢复, 备份恢复 rancher, rancher 备份与恢复 rancher] +--- + + + + + +在本节中，你将学习如何创建 Rancher 的备份，如何从备份中恢复 Rancher，以及如何将 Rancher 迁移到新的 Kubernetes 集群。 + +`rancher-backup` operator 可以用来备份和恢复任何 Kubernetes 集群上的 Rancher。这个应用是一个 Helm Chart，可以通过 Rancher 的 **Apps** 页面或使用 Helm CLI 部署。你可以访问[本页面](https://github.com/rancher/charts/tree/release-v2.6/charts/rancher-backup)获取 `rancher-backup` Helm Chart。 + +backup-restore operator 需要安装在 local 集群上，并且只对 Rancher 应用进行备份。备份和恢复操作仅在本地 Kubernetes 集群中执行。 + + +## 备份和恢复 Docker 安装的 Rancher + +对于使用 Docker 安装的 Rancher，请参见[备份](back-up-docker-installed-rancher.md)和[恢复](restore-docker-installed-rancher.md)对 Rancher 进行备份和恢复。 + +## 备份和恢复原理 + +`rancher-backup` operator 引入了三个自定义资源，分别是 Backups、Restores 和 ResourceSets。将以下集群范围的自定义资源定义添加到集群中： + +- `backups.resources.cattle.io` +- `resourcesets.resources.cattle.io` +- `restores.resources.cattle.io` + +ResourceSet 定义了需要备份哪些 Kubernetes 资源。由于备份 Rancher 所需的值是预设的，因此 ResourceSet 无法通过 Rancher UI 进行配置。请不要修改此 ResourceSet。 + +在创建 Backup 自定义资源时，`rancher-backup` operator 调用 `kube-apiserver` 来获取 Backup 自定义资源引用的 ResourceSet（即预设的 `rancher-resource-set`）资源。 + +然后，operator 以 .tar.gz 格式创建备份文件，并将其存储在 Backup 资源中配置的位置。 + +在创建 Restore 自定义资源时，operator 访问 Restore 指定的 tar.gz 备份文件，并从该文件恢复应用。 + +你可以使用 Rancher UI 或 `kubectl apply` 来创建 Backup 和 Restore 自定义资源。 + +:::note + +请参见[此处](migrate-rancher-to-new-cluster.md#2-使用-restore-自定义资源来还原备份)获取在 Rancher 2.6.3 中将现有备份文件恢复到 v1.22 集群的帮助。 + +::: + +## 安装 rancher-backup operator + +你可以使用 Rancher UI 或 Helm CLI 来安装 `rancher-backup` operator。两种安装方法都将 `rancher-backup` Helm Chart 安装在运行 Rancher Server 的 Kubernetes 集群上。它是集群管理员独有的功能，仅适用于 **local** 集群。（*如果你在 Rancher UI 中没有看到 `rancher-backup`，你可能选择了错误的集群。*） + +:::note + +使用 backup-restore-operator 执行恢复后，Fleet 中会出现一个已知问题：用于 clientSecretName 和 helmSecretName 的密文不包含在 Fleet 的 Git 仓库中。请参见[此处](../../../integrations-in-rancher/fleet/overview.md#故障排除)获得解决方法。 + +::: + +### 使用 Rancher UI 安装 rancher-backup + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面上，转到 `local` 集群并单击 **Explore**。 +1. 在左侧导航栏中，单击 **Apps > Charts**。 +1. 点击 **Rancher 备份**。 +1. 单击**安装**。 +1. 可选：配置默认存储位置。如需获取帮助，请参见[配置](../../../reference-guides/backup-restore-configuration/storage-configuration.md)。 +1. 单击**安装**。 + +**结果**：`rancher-backup` operator 已安装。 + +在**集群仪表板**中，你可以看到列在 **Deployments** 下的 `rancher-backup` operator。 + +如果需要在 Rancher 中配置备份应用，在左侧导航栏中单击 **Rancher 备份**。 + +### RBAC + +只有 Rancher 管理员和本地集群的所有者可以： + +* 安装 Chart +* 看到 Backup 和 Restore CRD 的导航链接 +* 通过分别创建 Backup CR 和 Restore CR 执行备份和恢复 +* 列出目前已执行的备份和恢复操作 + +## 备份 Rancher + +备份是通过创建 Backup 自定义资源实现的。如需查看教程，请参见[本页面](back-up-rancher.md)。 + +## 还原 Rancher + +还原是通过创建 Restore 自定义资源实现的。如需查看教程，请参见[本页面](restore-rancher.md)。 + +## 将 Rancher 迁移到新集群 + +你可以按照[这些步骤](migrate-rancher-to-new-cluster.md)执行迁移。 + +## 默认存储位置配置 + +配置一个用户保存备份文件的默认存储位置。有多种选项可以配置，例如将与 S3 兼容的对象存储作为单个备份文件的存储后端，或在安装 `backup-restore-operator` Helm Chart 时选择一个现有的 `StorageClass`。你还可以选择在每次备份时覆盖配置的存储位置，但这仅限于使用与 S3 兼容的对象存储或 Minio 对象存储。 + +如需了解各个选项的配置，请参见[本页面](../../../reference-guides/backup-restore-configuration/storage-configuration.md)。 + +### 示例 YAML 文件：Rancher Backup Helm Chart + +当使用 Helm CLI 安装时，可以使用示例 [values.yaml 文件](../../../reference-guides/backup-restore-configuration/storage-configuration.md#rancher-backup-helm-chart-的示例-valuesyaml) 来配置 `rancher-backup-operator`。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md index ea3dafca61e..eeb6cb521f0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md @@ -4,38 +4,23 @@ title: 还原 Rancher 本页概述了如何使用 Rancher 执行恢复。 -:::note 重要提示： +在以下情况下，请按照本页中的说明进行操作： +- 正在运行的 Rancher 实例与备份时的版本相同。 +- 上游（本地）集群与备份的位置相同。 -* 请按照此页面上的说明在已备份的同一集群上还原 Rancher。要把 Rancher 迁移到新集群，请参照步骤[迁移 Rancher](migrate-rancher-to-new-cluster.md)。 -* 在使用相同设置还原 Rancher 时，operator 将在还原开始时缩减 Rancher deployment，还原完成后又会扩展 deployment。因此，Rancher 在还原期间将不可用。 +:::note 重要提示 + +在使用相同设置还原 Rancher 时，operator 将在还原开始时缩减 Rancher deployment，还原完成后又会扩展 deployment。因此，Rancher 在还原期间将不可用。 + +::: + +:::tip + +* 按照以下步骤[迁移 Rancher](migrate-rancher-to-new-cluster.md)。 * 如果你需要在升级后将 Rancher 还原到先前版本，请参见[回滚](../../../getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md)。 ::: -## 使用 Rancher 2.6.4+ 进行回滚的其他步骤 - -Rancher v2.6.4 将 cluster-api 模块从 v0.4.4 升级到 v1.0.2。反过来，cluster-api 的 v1.0.2 版本将集群 API 的自定义资源定义 (CRD) 从 `cluster.x-k8s.io/v1alpha4` 升级到 `cluster.x-k8s.io/v1beta1`。当你尝试将 Rancher v2.6.4 回滚到以前版本的 Rancher v2.6.x 时，CRD 升级到 v1beta1 会导致回滚失败。这是因为使用旧 apiVersion (v1alpha4) 的 CRD 与 v1beta1 不兼容。 - -要避免回滚失败，你需要在尝试恢复操作或回滚**之前**运行以下 Rancher 脚本： - -* `verify.sh`：检查集群中是否有任何与 Rancher 相关的资源。 -* `cleanup.sh`：清理集群。 - -有关详细信息和源代码，请参阅 [rancher/rancher-cleanup repo](https://github.com/rancher/rancher-cleanup)。 - -:::caution - -`cleanup.sh` 运行的时候会有停机时间，这是因为脚本会删除 Rancher 创建的资源。 - -::: - -### 从 v2.6.4+ 回滚到较低版本的 v2.6.x - -1. 按照[说明](https://github.com/rancher/rancher-cleanup/blob/main/README.md)运行脚本。 -1. 按照[说明](https://rancher.com/docs/rancher/v2.6/en/backups/migrating-rancher/)在现有集群上安装 rancher-backup Helm Chart 并恢复之前的状态。 - 1. 省略步骤 3。 - 1. 执行到步骤 4 时，在要回滚到的 local 集群上安装 Rancher 2.6.x 版本。 - ### 创建 Restore 自定义资源 还原是通过创建 Restore 自定义资源实现的。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md new file mode 100644 index 00000000000..4d4d465d136 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md @@ -0,0 +1,21 @@ +--- +title: 跨集群部署应用 +--- + + + + + +不同版本的 Rancher 提供了几种不同的方式来部署跨集群应用。 + +## Fleet + +Rancher v2.5 及更高版本使用 Fleet 跨集群部署应用 + +使用 Fleet 的持续交付是大规模的 GitOps。如需更多信息，请参阅 [Fleet](fleet.md)。 + +### 多集群应用 + +在 v2.5 之前的 Rancher 中，多集群应用功能用于跨集群部署应用。多集群应用功能已弃用，但仍可作为旧版功能使用。 + +详情请参阅[此文档](multi-cluster-apps.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/helm-charts-in-rancher/create-apps.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/helm-charts-in-rancher/create-apps.md index 554725a2e8e..0189de391a7 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/helm-charts-in-rancher/create-apps.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/helm-charts-in-rancher/create-apps.md @@ -2,16 +2,12 @@ title: 创建应用 --- -Rancher 的应用市场基于 Helm 仓库和 Helm Chart。你可以添加基于 HTTP 的标准 Helm 仓库以及任何包含 Chart 的 Git 仓库。 - :::tip 有关开发 Chart 的完整演示，请参阅 Helm 官方文档中的 [Chart 模板开发者指南](https://helm.sh/docs/chart_template_guide/)。 ::: - - ## Chart 类型 Rancher 支持两种不同类型的 Chart，分别是 Helm Chart 和 Rancher Chart。 @@ -108,17 +104,10 @@ rancher_max_version: 2.3.99 | variable | string | true | 定义 `values.yml` 文件中指定的变量名，嵌套对象使用 `foo.bar`。 | | label | string | true | 定义 UI 标签。 | | description | string | false | 指定变量的描述。 | -| type | string | false | 如果未指定，则默认为 `string`（支持的类型为 string、multiline、boolean、int、enum、password、storageclass、hostname、pvc 和 secret）。 | -| required | bool | false | 定义变量是否是必须的（true \| false）。 | +| type | string | false | 如果未指定，则默认为 `string`（支持的类型为 string、multiline、boolean、int、enum、password、storageclasshostname、pvc、secret 和 cloudcredential）。 | | default | string | false | 指定默认值。仅在 `values.yml` 文件中没有对应值时使用。 | | group | string | false | 按输入值对问题进行分组。 | -| min_length | int | false | 最小字符长度。 | -| max_length | int | false | 最大字符长度。 | -| min | int | false | 最小整数长度。 | -| max | int | false | 最大整数长度。 | | options | []string | false | 为 `enum` 类型的变量指定选项，例如：options:
- "ClusterIP"
- "NodePort"
- "LoadBalancer" | -| valid_chars | string | false | 输入字符验证的正则表达式。 | -| invalid_chars | string | false | 无效输入字符验证的正则表达式。 | | subquestions | []subquestion | false | 添加一组子问题。 | | show_if | string | false | 如果条件变量为 true，则显示当前变量。例如 `show_if: "serviceType=Nodeport"` | | show\_subquestion_if | string | false | 如果为 true 或等于某个选项，则显示子问题。例如 `show_subquestion_if: "true"` | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md new file mode 100644 index 00000000000..48ecc783bee --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md @@ -0,0 +1,216 @@ +--- +title: Helm Charts 和 Apps +--- + + + + + +在本节中，你将学习如何在 Rancher 中管理 Helm Chart 仓库和应用。 + +## Helm Charts 在 Rancher 中的工作原理 + +在 Rancher 中 Helm chart 仓库是使用 **Apps** 进行管理的。 + +Rancher 使用应用商店系统导入一系列的 charts 包到仓库里，然后使用这些 charts 来部署自定义的 Kubernetes 应用程序或 Rancher 的工具，如监控（Monitoring）或 Istio。Rancher 工具预先集成进了仓库，并通过独立的 Helm chart 进行部署。如果你有自己的额外仓库只需要添加到当前的集群中就可以正常部署。 + +### Catalogs, Apps, Rancher UI + +[在 Rancher v2.4 及更早版本中](/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md), 存储部署的应用程序的仓库被称为 "catalogs"。这些仓库是通过 UI 的 **Catalogs** 进行管理的。 + +Rancher v2.5 用新的 **Apps & Marketplace** 功能替代了之前的 **应用商店**。 + +从 Rancher v2.6.5 开始, **Apps & Marketplace** 功能在 UI 中被命名为 **Apps**。 + +### Versioning Scheme + +Rancher chart 的版本设计方案主要围绕 charts 的主要版本以及与上游 charts 相关的 `+up` 注释进行设计。 + +**主要版本:** charts 的主要版本与 Rancher 的特定次要版本相关联。当你升级到新的 Rancher 次要版本时，你应该确保所有的功能 charts 也被升级对应的主版本。 + +**基于上游的 Charts:** 在升级时，确保上游 chart 的版本与你的 Rancher 版本兼容。chart 的 `+up` 注释指示 Rancher chart 正在跟踪的上游版本。例如，`100.x.x+up16.6.0` 的监控（Monitoring）跟踪上游的 kube-prometheus-stack `16.6.0` 并附带一些额外的 Rancher 补丁。 + +在升级 Rancher 版本时，不要降级你正在使用的 chart 版本。例如，如果你在 Rancher v2.5 中使用的监控版本比 `16.6.0` 还要高，你不应该升级到 `100.x.x+up16.6.0`，而是应该在下一个发布中升级到适当的版本。 + +#### 预发布版本 + +预发布版本遵循[语义化版本 2.0.0 ](https://semver.org/)的[规范](https://semver.org/#spec-item-9)。 例如，版本为 `0.1.3-dev.12ab4f` 的 Helm chart 被认为是一个预发布版本。预发布版本默认不显示，必须进行配置以显示。 + +如何显示预发布版本： + +1. 点击右上角的用户头像。 +2. 点击 **Preferences**. +3. 在 **Helm Charts** 下, 选择 **Include Prerelease Versions**. + +### 版本控制方案 + +| **Name** | **支持的最低版本** | **支持的最高版本** | +| ---------------- | ------------ | ------------ | +| external-ip-webhook | 100.0.0+up1.0.0 | 100.0.1+up1.0.1 | +| harvester-cloud-provider | 100.0.2+up0.1.12 | 100.0.2+up0.1.12 | +| harvester-csi-driver | 100.0.2+up0.1.11 | 100.0.2+up0.1.11 | +| neuvector | 100.0.0+up2.2.0 | 100.0.0+up2.2.0 | +| rancher-alerting-drivers | 100.0.0 | 100.0.2 | +| rancher-backup | 2.0.1 | 2.1.2 | +| rancher-cis-benchmark | 2.0.1 | 2.0.4 | +| rancher-gatekeeper | 100.0.0+up3.6.0 | 100.1.0+up3.7.1 | +| rancher-istio | 100.0.0+up1.10.4 | 100.3.0+up1.13.3 | +| rancher-logging | 100.0.0+up3.12.0 | 100.1.2+up3.17.4 | +| rancher-longhorn | 100.0.0+up1.1.2 | 100.1.2+up1.2.4 | +| rancher-monitoring | 100.0.0+up16.6.0 | 100.1.2+up19.0.3 | +| rancher-sriov (experimental) | 100.0.0+up0.1.0 | 100.0.3+up0.1.0 | +| rancher-vsphere-cpi | 100.3.0+up1.2.1 | 100.3.0+up1.2.1 | +| rancher-vsphere-csi | 100.3.0+up2.5.1-rancher1 | 100.3.0+up2.5.1-rancher1 | +| rancher-wins-upgrader | 0.0.100 | 100.0.1+up0.0.1 | + +## 访问 Charts + +**Charts** 页面包含所有 Rancher、Rancher 合作伙伴和自定义 Chart。你可以通过选择左侧的下拉菜单来筛选 Chart： + +* Rancher 工具（例如 Logging 或 Monitoring）包含在 **Rancher** 标签下 +* Partner Chart 位于 **Partner** 标签下 +* 自定义 Chart 将显示在仓库的名称下 + +所有这三种类型都以相同的方式部署和管理。 + +:::note +由 Cluster Manager （旧版 Rancher UI 中的全局视图）管理的应用应继续仅由 Cluster Manager 管理，而在新 UI 中使用 Apps 管理的应用则仅能由 Apps 管理。 +::: + +访问 **Charts** 页面: + +1. 点击 **☰ > Cluster Management**。 +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中, 点击 **Apps > Charts**。 + +### 管理仓库 + +**Repositories** 页面列出了你的 Helm 仓库。 这包括传统的具有 index.yaml 的 Helm 端点，以及克隆并指向特定分支的 Git 仓库。要使用自定义 Charts，在这里可以添加你的仓库。添加仓库后，你可以在 **Charts** 页面中访问自定义 Charts，这些 Charts 将列在仓库的名称下。 + +访问 **Repositories** 页面: + +1. 点击 **☰ > Cluster Management**. +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 + +### 添加自定义 Git 仓库 + +要添加一个包含你的 Helm Charts 或集群模板定义的自定义 Git 仓库： + +1. 点击 **☰ > Cluster Management**。 +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 +4. 点击 **Create**。 +5. 选择 **Git repository containing Helm chart...**。 +6. 你必须输入名称和 Git 仓库的 URL。其他配置项包括描述，都是可选的。如果你不想设置默认的分支，可以输入你想要使用的分支名称。通常，默认分支名为 `main` 或 `master`。 +7. 点击 **Create** 添加。 + +在 Rancher 中添加 Charts 仓库后，它将立即生效。 + +### 添加自定义 Helm Chart 仓库 + +你可以将自己的 Helm chart 仓库添加到 Rancher。为了能正确添加 http 的 Helm Chart 仓库，你需要提供 chart 的服务器并能够响应 GET 请求并提供 YAML 文件和 tar 包。 + +有关 Helm chart 仓库的更多信息，请参阅 [官方 Helm 文档](https://helm.sh/docs/topics/chart_repository/)。 + +要将自定义 Helm chart 仓库添加到 Rancher： + +1. 点击 **☰ > Cluster Management**。 +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 +4. 点击 **Create**。 +5. 选择 **http(s) URL to an index generated by Helm**. +6. 输入仓库名称和 chart 的 index URL 地址。 +7. 点击 **Create** 添加。 + +### 添加私有 Git/Helm Chart 仓库 + +你可以使用 SSH 密钥凭据或 HTTP 基础认证秘密（如用户名和密码）添加私有 Git 或 Helm chart 仓库。 + +### 向仓库添加私有 CA + +向 Helm chart 仓库添加私有 CA，你必须将 DER 格式的 CA 证书的 base64 编码副本添加到 Chart 仓库的 `spec.caBundle 字段`，例如 `openssl x509 -outform der -in ca.pem | base64 -w0`。无论是基于 Git 还是 HTTP 的仓库，操作步骤都是相同的 + +1. 点击 **☰**。在左侧导航菜单的 **Explore Cluster**, 选择一个集群。 +2. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 +3. 找到你想要向其添加私有 CA 证书的 Git 或 HTTP 的仓库。点击 **⋮ > Edit YAML**。 +4. 设置 `caBundle` 值，如以下示例： + + ```yaml + [...] + spec: + caBundle: + MIIFXzCCA0egAwIBAgIUWNy8WrvSkgNzV0zdWRP79j9cVcEwDQYJKoZIhvcNAQELBQAwPzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRQwEgYDVQQKDAtNeU9yZywgSW5jLjENMAsGA1UEAwwEcm9vdDAeFw0yMTEyMTQwODMyMTdaFw0yNDEwMDMwODMyMT + ... + nDxZ/tNXt/WPJr/PgEB3hQdInDWYMg7vGO0Oz00G5kWg0sJ0ZTSoA10ZwdjIdGEeKlj1NlPyAqpQ+uDnmx6DW+zqfYtLnc/g6GuLLVPamraqN+gyU8CHwAWPNjZonFN9Vpg0PIk1I2zuOc4EHifoTAXSpnjfzfyAxCaZsnTptimlPFJJqAMj+FfDArGmr4= + [...] + ``` + +:::note 带有认证的 Helm chart 仓库 + +Repo.Spec 包含一个 `disableSameOriginCheck` 值，该值允许用户绕过相同源的检查，将仓库身份认证信息作为基本 Auth 标头与所有 API 调用一起发送。不建议采用这种做法，但这可以用作非标准 Helm Chart 仓库（例如重定向到不同源 URL 的仓库）的临时解决方案。 + +要将此功能用于现有 Helm Chart 仓库，请按照前面的步骤编辑 YAML。在 YAML 文件的 `spec` 部分，添加 `disableSameOriginCheck` 并将其设置为 `true`： + +```yaml +[...] +spec: + disableSameOriginCheck: true +[...] +``` + +::: + +### Helm 兼容性 + +仅支持 Helm 3 兼容 Chart 。 + +### 部署和升级 Chart + +安装和部署 chart: + +1. 点击 **☰ > Cluster Management**。 +2. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +3. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Charts**。 +4. 选择一个 chart 点击 **Install**。 + +Rancher 和 Partner Chart 可能通过自定义页面或 questions.yaml 文件进行额外的配置，但所有 Chart 安装都可以修改 values.yaml 和其他基本设置。单击安装后，将部署一个 Helm 操作作业，并显示该作业的控制台。 + +要查看所有最近的更改，点击左侧导航栏菜单中的 **APPs > Recent Operations**。你可以查看已进行的调用、条件、事件和日志 + +安装 Chart 后，点击左侧导航栏菜单中的 **Apps > Installed Apps**。在本节中，你可以升级或删除安装，并查看更多详细信息。选择升级时，呈现的形式和数值与安装相同。 + +大多数 Rancher 工具在 **Apps** 下方的工具栏中都有额外的页面，以帮助你管理和使用这些功能。这些页面包括指向仪表板的链接、可轻松添加自定义资源的表单以及其他信息。 + +:::caution + +如果在升级前使**自定义 Helm 选项**，并且你的 Chart 中包含不可更改的字段，使用 `--force` 选项可能会导致错误。这是因为 Kubernetes 中的某些对象一旦创建就无法更改。要避免该错误，你可以： + +* 使用默认升级选项（即不要使用 `--force` 选项） +* 卸载现有 Chart 并安装升级后的 Chart +* 在执行强制升级之前删除集群中具有不可更改字段的资源 + +::: + +#### 旧版应用 + +在 **Apps > Installed Apps** 页面上，旧版应用没有升级按钮。 + +如果你想升级已安装的旧版应用， 必须启用[旧版功能](../../advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 如果你在升级 Rancher 之前已经运行了旧版应用，此标志会自动启用。 + +1. 如果尚未启用，启用[旧版功能](../../advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 +2. 点击 **☰ > Cluster Management**。 +3. 找到你想要访问 Charts 的集群名称。点击集群行末尾的 **Explore**。 +4. 点击 **Legacy > Project > Apps**. + +如果在 **Legacy > Project** 下看不到 **Apps** ,请点击顶部导航栏的 **project/namespace** 搜索栏，并从下拉菜单中选择相关项目。 + +要升级旧版多集群应用： + +1. 点击 **☰**. +2. 在 **Legacy Apps** 下点击 **Multi-cluster Apps**. + + +### 限制 + +Rancher CLI 不能用于安装仪表板应用程序或 Rancher 功能 Chart。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md new file mode 100644 index 00000000000..ee2843a4bd6 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/infrastructure-setup/infrastructure-setup.md @@ -0,0 +1,13 @@ +--- +title: Kubernetes 集群基础设施 +--- + + + + + +还没有适用于 Kubernetes 集群的基础架构？请尝试以下教程之一。 + +要为具有外部数据库的高可用 K3s Kubernetes 集群设置基础设施，请参见[本页面](ha-k3s-kubernetes-cluster.md)。 + +要为高可用 RKE Kubernetes 集群设置基础设施，请参见[本页面](ha-rke1-kubernetes-cluster.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md new file mode 100644 index 00000000000..f8213ffbbac --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-cluster-setup/kubernetes-cluster-setup.md @@ -0,0 +1,11 @@ +--- +title: "Kubernetes 使用教程" +--- + + + + + +本章节介绍如何安装 Kubernetes 集群，使得 Rancher Server 可以安装在该集群上。 + +Rancher 可以在任何 Kubernetes 集群上运行。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md new file mode 100644 index 00000000000..313585d60dd --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md @@ -0,0 +1,52 @@ +--- +title: 生产就绪集群检查清单 +--- + + + + + +本节将介绍创建生产就绪型 Kubernetes 集群的最佳实践。这个集群可用于运行你的应用和服务。 + +有关集群的要求（包括对 OS/Docker、硬件和网络的要求），请参阅[节点要求](../node-requirements-for-rancher-managed-clusters.md)部分。 + +本文介绍了我们推荐用于所有生产集群的最佳实践的简短列表。 + +如需获取推荐的所有最佳实践的完整列表，请参阅[最佳实践](../../../../reference-guides/best-practices/best-practices.md)部分。 + +### 节点要求 + +- 确保你的节点满足所有[节点要求](../node-requirements-for-rancher-managed-clusters.md)，包括端口要求。 + +### 备份 etcd + +* 启用 etcd 快照。验证是否正在创建快照，并执行灾难恢复方案，从而验证快照是否有效。etcd 是存储集群状态的位置，丢失 etcd 数据意味着丢失集群。因此，请确保为集群配置 etcd 的定期快照，并确保快照也是存储在外部（节点外）的。 + +### 集群架构 + +* 节点应具有以下角色配置之一： + * `etcd` + * `controlplane` + * `etcd` 和 `controlplane` + * `worker`（不应在具有 `etcd` 或 `controlplane` 角色的节点上使用或添加 `worker` 角色） +* 至少拥有三个角色为 `etcd` 的节点，来确保失去一个节点时仍能存活。增加 etcd 节点数量能提高容错率，而将 etcd 分散到不同可用区甚至能获取更好的容错能力。 +* 为两个或更多节点分配 `controlplane` 角色，能实现主组件的高可用性。 +* 为两个或多个节点分配 `worker` 角色，以便在节点故障时重新安排工作负载。 + +有关每个角色的用途的更多信息，请参阅 [Kubernetes 中的节点角色](roles-for-nodes-in-kubernetes.md)。 + +有关每个 Kubernetes 角色的节点数的详细信息，请参阅[推荐架构](../../../../reference-guides/rancher-manager-architecture/architecture-recommendations.md)部分。 + +### Logging 和 Monitoring + +* 为 Kubernetes 组件（系统服务）配置告警/通知程序。 +* 为集群分析和事后剖析配置 Logging。 + +### 可靠性 + +* 在集群上执行负载测试，以验证硬件是否可以支持你的工作负载。 + +### 网络 + +* 最小化网络延迟。Rancher 建议尽量减少 etcd 节点之间的延迟。`heartbeat-interval` 的默认设置是 `500`，`election-timeout` 的默认设置是 `5000`。这些 [etcd 调优设置](https://coreos.com/etcd/docs/latest/tuning.html) 允许 etcd 在大多数网络（网络延迟特别高的情况下除外）中运行。 +* 集群节点应位于单个区域内。大多数云厂商在一个区域内提供多个可用区，这可以提高你集群的可用性。任何角色的节点都可以使用多个可用区。如果你使用 (../set-up-cloud-providers/set-up-cloud-providers.md) 资源，请查阅文档以了解限制（即区域存储限制）。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md new file mode 100644 index 00000000000..44c89d5068d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md @@ -0,0 +1,78 @@ +--- +title: Rancher 中的 Kubernetes 集群设置 +description: 配置 Kubernetes 集群 +--- + + + + + +Rancher 简化了集群的创建，允许你通过 Rancher UI 而不是更复杂的替代方案来创建集群。Rancher 为启动集群提供了多个选项。 请选择最适合您用例的选项。 + +本节假定你对 Docker 和 Kubernetes 有基本的了解。有关 Kubernetes 组件如何协同工作的简要说明，请参阅[概念](../../../reference-guides/kubernetes-concepts.md)页面。 + +有关 Rancher 服务如何配置集群以及使用哪些工具配置集群的概念性概述，请参阅[架构](../../../reference-guides/rancher-manager-architecture/rancher-manager-architecture.md)页面。 + +### 按集群类型划分的集群管理功能 + +下表总结了每种集群类型的可用选项和设置： + +import ClusterCapabilitiesTable from '../../../shared-files/\_cluster-capabilities-table.md'; + + + +## 在托管 Kubernetes 提供商中设置集群 + +在这种情况下，Rancher 不会配置 Kubernetes，因为它是由 Google Kubernetes Engine（GKE）、Amazon Elastic Container Service for Kubernetes 或 Azure Kubernetes Service 等提供商安装的。 + +如果使用 Google GKE 等 Kubernetes 提供商，Rancher 会与其云 API 集成，允许你从 Rancher UI 为托管集群创建和管理基于角色的访问控制。 + +更多信息，请参阅[托管 Kubernetes 集群](set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md)部分。 + +## 使用 Rancher 启动 Kubernetes + +在自己的节点上配置 Kubernetes 时，Rancher 使用 [Rancher Kubernetes Engine (RKE)](https://rancher.com/docs/rke/latest/en/) 作为库。RKE 是 Rancher 自己的轻量级 Kubernetes 安装程序。 + +在 RKE 集群中，Rancher 负责管理 Kubernetes 的部署。这些集群可以部署在任何裸机服务器、云提供商或虚拟化平台上。 + +这些节点可以通过 Rancher 的用户界面动态配置，它可以调用 [Docker Machine](https://docs.docker.com/machine/) 在各种云提供商上启动节点。 + +如果你已经有了一个要添加到 RKE 集群的节点，可以通过在该节点上运行 Rancher 代理容器将其添加到集群中。 + +更多信息，请参阅 [RKE 集群](../launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)部分。 + +### 在基础设施提供商中启动 Kubernetes 和配置节点 + +Rancher 可以在亚马逊 EC2、DigitalOcean、Azure 或 vSphere 等基础设施提供商中动态调配节点，然后在这些节点上安装 Kubernetes。 + +使用 Rancher，你可以基于[节点模板](../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md#node-templates)创建节点池。该模板定义了用于在云提供商中启动节点的参数。 + +使用基础设施提供商托管的节点的一个好处是，如果某个节点失去了与集群的连接，Rancher 可以自动替换掉它，从而保持预期的集群配置。 + +可用于创建节点模板的云提供商是由 Rancher UI 中激活的[节点驱动程序](../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md#node-drivers)决定的。 + +有关详细信息，请参阅[基础设施提供商托管的节点](../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)部分。 + +### 在已存在的自定义节点上启动 Kubernetes + +设置这种集群时，Rancher 会在已存在的[自定义节点](../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)上安装 Kubernetes，从而创建一个自定义集群。 + +你可以将任何节点添加到 Rancher 中，并用它们创建集群。 + +这些节点包括内部部署的裸机服务器、云托管的虚拟机或内部部署的虚拟机。 + +## 注册已存在的集群 + +集群注册功能取代了集群导入功能。 + +现在，注册 EKS 集群还能带来更多好处。在 Rancher UI 中，注册的 EKS 集群和在 Rancher 中创建的 EKS 集群在大多数情况下的处理方式相同，但删除除外。 + +删除在 Rancher 中创建的 EKS 集群时，集群将被销毁。删除在 Rancher 中注册的EKS集群时，该集群会断开与 Rancher 服务器的连接，但它仍然存在，你仍然可以按照在 Rancher 中注册前的方式访问它。 + +更多信息，请参阅[本页](register-existing-clusters.md)。 + +## 以编程方式创建集群 + +通过 Rancher 以编程方式部署 Kubernetes 集群的最常用方法是使用 Rancher2 Terraform 提供商。使用 Terraform 创建集群的文档在[这里](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster)。 + +可以使用 Terraform 创建或导入 EKS、GKE、AKS 集群和 RKE 集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-amazon.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-amazon.md new file mode 100644 index 00000000000..054fd82accd --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-amazon.md @@ -0,0 +1,198 @@ +--- +title: 将 Amazon 从树内迁移到树外 +--- + + + + + +:::note +Rancher Prime 提供的 Rancher v2.7.11，是 Rancher v2.7.x 系列支持 Kubernetes v1.27 的版本。如果你使用 Rancher v2.7.11 并将 Kubernetes 集群升级到了 1.27，则必须使用树外的云提供商。 +::: + +Kubernetes 不再在树内维护云提供商。在 Kubernetes v1.27 及更高版本中，树内云提供商已被移除。当你从树内提供商迁移到树外提供商时，Rancher UI 允许你升级到 Kubernetes v1.27。 + +不过，如果你执行的是手动迁移，现有集群必须在迁移后升级到 Kubernetes v1.27 才能继续运行。 + +要从树内云提供商迁移到树外 AWS 云提供商，必须停止现有集群的 kube 控制器管理器，并安装 AWS 云控制器管理器。有许多方法可以做到这一点。有关详情，请参阅有关[外部云控制器管理器](https://cloud-provider-aws.sigs.k8s.io/getting_started/)的 AWS 官方文档。 + +如果可以接受迁移过程中出现一些停机，请按照说明[设置外部云提供商](../set-up-cloud-providers/amazon.md#using-the-out-of-tree-aws-cloud-provider)。这些说明概述了如何为新配置的集群配置树外云提供商。在设置过程中，会有一些停机，因为从旧云提供商停止运行到新云提供商开始运行之间会有一段时间的间隔。 + +如果您的设置不能容忍任何控制平面停机，则必须启用领导者迁移。这有助于从 kube 控制器管理器中的控制器顺利过渡到云控制器管理器中的对应控制器。有关详细信息，请参阅 AWS 官方文档[使用领导者迁移](https://cloud-provider-aws.sigs.k8s.io/getting_started/)。 + +:::note Important: +Kubernetes [云控制器迁移文档](https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/#before-you-begin)指出，可以使用相同的 Kubernetes 版本进行迁移，但假设迁移是 Kubernetes 升级的一部分。请参考有关[迁移到要使用的云控制器管理器](https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/)的 Kubernetes 文档，了解迁移前是否需要自定义设置。确认[迁移配置值](https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/#default-configuration)。如果您的云提供商提供 Node IPAM 控制器的实现，您还需要迁移 [IPAM 控制器](https://kubernetes.io/docs/tasks/administer-cluster/controller-manager-leader-migration/#node-ipam-controller-migration)。 +::: + + + + +1. 更新集群配置，启用领导者迁移： + +```yaml +spec: + rkeConfig: + machineSelectorConfig: + - config: + kube-controller-manager-arg: + - enable-leader-migration + machineLabelSelector: + matchExpressions: + - key: rke.cattle.io/control-plane-role + operator: In + values: + - "true" +``` + +请注意，云提供商在此步骤中仍是`aws` + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + cloud-provider-name: aws +``` + +2. 标记控制平面节点不可调度，以便 AWS 云控制器 pod 仅在升级到外部云提供商后才在节点上运行： + +```shell +kubectl cordon -l "node-role.kubernetes.io/control-plane=true" +``` + +3. 要安装启用了领导者迁移的 AWS 云控制器管理器，请遵循[部署云控制器管理器 Chart](../set-up-cloud-providers/amazon.md#using-the-out-of-tree-aws-cloud-provider) 的步骤 1-3。从 Kubernetes 1.22 起，kube-controller-manager 将使用默认配置，该配置将满足 controller-to-manager 的迁移。更新 `spec.rkeConfig.additionalManifest` 下 `aws-cloud-controller-manager` 的容器参数，以启用领导者迁移： + +```shell +- '--enable-leader-migration=true' +``` + +4. 安装 Chart 并确认 Daemonset `aws-cloud-controller-manager` 已成功部署： + +```shell +kubectl rollout status daemonset -n kube-system aws-cloud-controller-manager +``` + +5. 更新配置集群以更改云提供商，并从 kube 控制器中移除领导者迁移 参数。如果升级 Kubernetes 版本，也要在集群 YAML 文件的 `spec.kubernetesVersion` 部分设置 Kubernetes 版本 + +:::note Important + +如果不依赖 rke2 监管程序正确设置提供商 ID，则只移除 `cloud-provider-name: aws`。 + +::: + +如果不想在集群中启用 `enable-leader-migration` 功能，请将其移除： + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + cloud-provider-name: external +``` + +移除 `enable-leader-migration`： + +```yaml +spec: + rkeConfig: + machineSelectorConfig: + - config: + kube-controller-manager-arg: + - enable-leader-migration + machineLabelSelector: + matchExpressions: + - key: rke.cattle.io/control-plane-role + operator: In + values: + - "true" +``` + +:::tip + +您也可以在升级后禁用领导者迁移，因为由于只有一个云控制器管理器，不再需要领导者迁移，可以将其移除。升级 Chart 并从容器参数中删除以下部分： + +```yaml +- --enable-leader-migration=true +``` + +::: + +使用以下命令验证云控制器管理器更新是否成功上线： + +```shell +kubectl rollout status daemonset -n kube-system aws-cloud-controller-manager +``` + +6. 云提供商负责设置节点的 ProviderID。检查是否所有节点都用 ProviderID 进行了初始化： + +```shell +kubectl describe nodes | grep "ProviderID" +``` + + + + + +1. 更新集群配置，在 `cluster.yml` 中启用领导者迁移： + +```yaml +services: + kube-controller: + extra_args: + enable-leader-migration: "true" +``` + +请注意，云提供商在此步骤中仍是`aws` + +```yaml +cloud_provider: + name: aws +``` + +2. 标记控制平面节点不可调度，以便 AWS 云控制器 pod 仅在升级到外部云提供商后才在节点上运行： + +```shell +kubectl cordon -l "node-role.kubernetes.io/controlplane=true" +``` + +3. 要安装 AWS 云控制器管理器，必须启用领导者迁移，并遵循在新集群上安装 AWS 时的相同步骤。要启用领导者迁移，请在步骤 7 中的容器参数中添加以下内容，同时[按照步骤安装 Chart](../set-up-cloud-providers/amazon.md#helm-chart-installation-from-ui)： + +```yaml +- "--enable-leader-migration=true" +``` + +4. 确认 Chart 已安装，但由于控制面板节点不可调度，新的 pod 尚未运行。在下一步更新集群后，RKE 将升级并允许对每个节点的调度，并调度 `aws-controller-manager` pod。 + +5. 更新 `cluster.yml`以更改云提供商，并从 kube-controller 中移除领导者迁移参数。 + +选择 **External Amazon (out-of-tree)** 可设置 `--cloud-provider=external`，并启用 `useInstanceMetadataHostname`。节点驱动集群和自定义集群必须启用 `useInstanceMetadataHostname`，否则无法通过 `--node-name`提供自定义节点名称。启用 `useInstanceMetadataHostname` 会查询 ec2 元数据服务，并为 `kubelet` 和 `kube-proxy`将 `/hostname` 设置为 `hostname-override` ： + +```yaml +rancher_kubernetes_engine_config: + cloud_provider: + name: external-aws + useInstanceMetadataHostname: true/false +``` + +如果不想在集群中启用 `enable-leader-migration`，请移除它： + +```yaml +services: + kube-controller: + extra_args: + enable-leader-migration: "true" +``` + +:::tip +完成迁移后，还可以禁用领导者迁移。升级 Chart 并从容器参数中删除以下部分： + +```yaml +- --enable-leader-migration=true +``` + +::: + +6. 如果要升级集群的 Kubernetes 版本，也要设置 Kubernetes 版本。 + +7. 更新集群。现在，`aws-cloud-controller-manager` pod 应该已经运行。 + + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md new file mode 100644 index 00000000000..760bde3919e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md @@ -0,0 +1,115 @@ +--- +title: 将 vSphere 从树内迁移到树外 +--- + + + + + +:::note +Rancher Prime 提供的 Rancher v2.7.11，是 Rancher v2.7.x 系列支持 Kubernetes v1.27 的版本。如果你使用 Rancher v2.7.11 并将 Kubernetes 集群升级到了 1.27，则必须使用树外的云提供商。 +::: + +Kubernetes 不再在树内维护云提供商。vSphere 有一个树外云提供商，可以通过安装 vSphere 云提供商和云存储插件来使用。 + +本页面介绍如何从树内 vSphere 云提供商迁移到树外，并在迁移后管理现有虚拟机。 + +它遵循官方 [vSphere 迁移文档](https://vsphere-csi-driver.sigs.k8s.io/features/vsphere_csi_migration.html)中提供的步骤，并提供在 Rancher 中执行的步骤。 + +### Cloud-config 格式限制 + +由于 vSphere Cloud Storage Interface (CSI) 中的一个现有错误，使用以下 cloud-config 格式配置的现有卷将无法迁移。 + +如果 cloud-config 中的数据存储和资源池路径采用这种格式，vsphere CSI 驱动程序将无法识别它： + +```yaml +default-datastore: /datastore/ +resourcepool-path: "/host//Resources/" +``` + +使用树内提供商以下列格式配置的卷将正确迁移： + +```yaml +default-datastore: +resourcepool-path: "/Resources/" +``` + +上游 bug: https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/628 + +Rancher 问题跟踪此 bug: https://github.com/rancher/rancher/issues/31105 + +## 前提条件 + +- vSphere CSI 迁移需要 vSphere 7.0u1。为了能够管理现有树内 vSphere 卷，请将 vSphere 升级到 7.0u1。 +- Kubernetes 版本必须为 1.19 或更高。 + +## 迁移 + +### 1. 安装 CPI 插件 + +安装 CPI 之前，我们需要用 `node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule` 标记所有节点。 + +可以通过运行以下命令来完成： + +``` +curl -O https://raw.githubusercontent.com/rancher/helm3-charts/56b622f519728378abeddfe95074f1b87ab73b1e/charts/vsphere-cpi/taints.sh +``` + +或: + +``` +wget https://raw.githubusercontent.com/rancher/helm3-charts/56b622f519728378abeddfe95074f1b87ab73b1e/charts/vsphere-cpi/taints.sh +chmod +x taints.sh +./taints.sh +``` + +运行脚本并标记所有节点后，启动 Helm vSphere CPI Chart。 + +1. 点击 **☰ > Cluster Management**. +1. 转到将要安装 vSphere CPI Chart 的集群，然后点击 **Explore**. +1. 点击 **Apps > Charts**. +1. 点击 **vSphere CPI**. +1. 点击 **Install**. +1. 填写所需的 vCenter 详细信息，然后单击 **Install**. + +vSphere CPI 会使用 ProviderID 初始化所有节点，这是 vSphere CSI 驱动程序所需要的。 + +使用以下命令检查是否用 ProviderID 初始化了所有节点： + +``` +kubectl describe nodes | grep "ProviderID" +``` + +### 2. 安装 CSI 驱动程序 + +1. 点击 **☰ > Cluster Management**. +1. 转到将要安装 vSphere CSI Chart 的集群，点击 **Explore**. +1. 点击 **Apps > Charts**. +1. 点击 **vSphere CSI**. +1. 点击 **Install**. +1. 填写所需的 vCenter 详细信息，然后单击 **Install**. +1. 选中 **Customize Helm options before install**，然后单击 **Next**. +1. 在 **Features** 选项卡上， 选中 **Enable CSI Migration**. +1. （可选）转到 **Storage** 选项卡设置数据存储。此 Chart 会创建一个以 `csi.vsphere.vmware.com` 作为供应商的 StorageClass。在创建此 StorageClass 时，您可以提供用于 CSI 卷配置的数据存储的 URL。数据存储 URL 可在 vSphere 客户端中通过选择数据存储并转到摘要选项卡找到。填写 StorageClass 的详细信息。 +1. 点击 **Install**. + +### 3. 编辑集群以启用 CSI 迁移功能标志 + +1. 编辑集群时，如果 Kubernetes 版本低于 1.19，请从 **Kubernetes Version** 下拉菜单中选择 1.19 或更高版本。 +2. 要启用功能标志，请单击 "Edit as YAML", 然后在 kube-controller 和 kubelet 下添加以下内容: + + ```yaml + extra_args: + feature-gates: "CSIMigration=true,CSIMigrationvSphere=true" + ``` + +### 4. 清空工作节点 + +在升级过程中，必须先清空工作节点，然后再更改kubelet和kube-controller-manager参数。 + +1. 点击 **☰ > Cluster Management**。 +1. 转到要清空工作节点的集群，点击 **⋮ > Edit Config**。 +1. 在 **Advanced Options** 部分，将 **Maximum Worker Nodes Unavailable** 字段设为 1。 +1. 要在升级期间清空节点，请选择 **Drain Nodes > Yes**. +1. 将 **Force** 和 **Delete Local Data** 设置为 **true**. +1. 点击 **Save** 升级集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/google-compute-engine.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/google-compute-engine.md index 377a2929010..f5e437e9de8 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/google-compute-engine.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/google-compute-engine.md @@ -4,7 +4,7 @@ title: 设置 Google Compute Engine 云提供商 在本节中，你将了解如何在 Rancher 中为自定义集群启用 Google Compute Engine (GCE) 云提供商。自定义集群指的是 Rancher 在现有节点上安装 Kubernetes 的集群。 -详情请参见 [GCE 云提供商的官方 Kubernetes 文档](https://github.com/kubernetes/website/blob/release-1.18/content/en/docs/concepts/cluster-administration/cloud-providers.md#gce)。 +详情请参见 [GCE 云提供商的官方 Kubernetes 文档](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#gce)。 :::note 先决条件： diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md new file mode 100644 index 00000000000..f6aa2871a8c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md @@ -0,0 +1,47 @@ +--- +title: 设置云提供商 +--- + + + + + +_cloud provider_ 是 Kubernetes 中的一个模块，它提供了一个用于管理节点、负载均衡器和网络路由的接口。 + +在 Rancher 中设置 cloud provider 时，如果你使用的云提供商支持自动化，Rancher Server 可以在启动 Kubernetes 定义时自动配置新节点、负载均衡器或持久存储设备。 + +如果你配置的节点云提供商集群不满足先决条件，集群将无法正确配置。 + +**Cloud Provider** 选项默认设置为 `None`。 + +可以启用的云提供商包括： + +* Amazon +* Azure +* GCE (Google Compute Engine) +* vSphere + +### 设置 Amazon 云提供商 + +有关启用 Amazon 云提供商的详细信息，请参阅[此页面](amazon.md)。 + +### 设置 Azure 云提供商 + +有关启用 Azure 云提供商的详细信息，请参阅[此页面](azure.md)。 + +### 设置 GCE 云提供商 + +有关启用 Google Compute Engine 云提供商的详细信息，请参阅[此页面](google-compute-engine.md)。 + +### 设置 vSphere 云提供商 + +有关启用 vSphere 云提供商的详细信息，请参阅[树内 vSphere 配置](configure-in-tree-vsphere.md) 和 [树外 vSphere 配置](configure-out-of-tree-vsphere.md)。 + +### 设置自定义云提供商 + +任何 Kubernetes Cloud Provider 都可以通过`自定义`云提供商进行配置。 + +对于自定义云提供商选项，你可以参考 [RKE 文档](https://rancher.com/docs/rke/latest/en/config-options/cloud-providers/)，了解如何为你的云提供商编辑 yaml 文件。特定云提供商的详细配置说明如下： + +- [vSphere](https://rke.docs.rancher.com/config-options/cloud-providers/vsphere) +- [OpenStack](https://rancher.com/docs/rke/latest/en/config-options/cloud-providers/openstack/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md new file mode 100644 index 00000000000..dd034e3daf8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-clusters-from-hosted-kubernetes-providers/set-up-clusters-from-hosted-kubernetes-providers.md @@ -0,0 +1,33 @@ +--- +title: 通过托管 Kubernetes 提供商设置集群 +--- + + + + + +在这种情况下，Rancher 不会配置 Kubernetes，因为它是由 Google Kubernetes Engine (GKE)、Amazon Elastic Container Service for Kubernetes 或 Azure Kubernetes Service 等提供商安装的。 + +如果你使用 Kubernetes 提供商，例如 Google GKE，Rancher 将与对应的云 API 集成，允许你从 Rancher UI 为托管集群创建和管理 RBAC。 + +在这个用例中，Rancher 使用提供商的 API 向托管提供商发送请求。然后，提供商会为你配置和托管集群。集群创建成功后，你可以像管理本地集群或云上集群一样，通过 Rancher UI 对集群进行管理。 + +Rancher 支持以下 Kubernetes 提供商： + +- [Google GKE (Google Kubernetes Engine)](https://cloud.google.com/kubernetes-engine/) +- [Amazon EKS (Amazon Elastic Container Service for Kubernetes)](https://aws.amazon.com/eks/) +- [Microsoft AKS (Azure Kubernetes Service)](https://azure.microsoft.com/en-us/services/kubernetes-service/) +- [Alibaba ACK (Alibaba Cloud Container Service for Kubernetes)](https://www.alibabacloud.com/product/kubernetes) +- [Tencent TKE (Tencent Kubernetes Engine)](https://intl.cloud.tencent.com/product/tke) +- [Huawei CCE (Huawei Cloud Container Engine)](https://www.huaweicloud.com/en-us/product/cce.html) + +## 托管 Kubernetes 提供商的认证 + +使用 Rancher 创建由提供商托管的集群时，你需要输入认证信息。Rancher 会使用认证信息来访问云厂商的 API。有关如何获取此信息的详情，请参阅： + +- [创建 GKE 集群](gke.md) +- [创建 EKS 集群](eks.md) +- [创建 AKS 集群](aks.md) +- [创建 ACK 集群](alibaba.md) +- [创建 TKE 集群](tencent.md) +- [创建 CCE 集群](huawei.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md new file mode 100644 index 00000000000..09d36c82e7c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md @@ -0,0 +1,284 @@ +--- +title: 在 Windows 集群上启动 Kubernetes +--- + + + + + +使用 Rancher 配置[自定义集群](../../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)时，Rancher 通过 RKE（Rancher Kubernetes Engine）在现有节点上安装 Kubernetes。 + +在使用 Rancher 配置的 Windows 集群中，集群必须同时包含 Linux 和 Windows 节点。Kubernetes controlplane 只能运行在 Linux 节点上，Windows 节点只能有 Worker 角色。Windows 节点只能用于部署工作负载。 + +Windows 集群的其他要求如下： + +- 只有在创建集群时启用了 Windows 支持的集群才能添加 Windows 节点。无法为现有集群启用 Windows 支持。 +- 需要 Kubernetes 1.15+。 +- 必须使用 Flannel 网络提供商。 +- Windows 节点必须有 50 GB 的磁盘空间。 + +有关完整的要求列表，请参阅[本节](#windows-集群的要求)。 + +有关支持 Windows 的 Kubernetes 功能摘要，请参阅[在 Windows 中使用 Kubernetes 支持的功能和限制](https://kubernetes.io/docs/setup/production-environment/windows/intro-windows-in-kubernetes/#supported-functionality-and-limitations) 的 Kubernetes 文档，或[在 Kubernetes 中调度 Windows 容器的指南](https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-containers/)。 + +### RKE2 Windows + +RKE2 配置功能还包括在 Windows 集群上安装 RKE2。RKE2 的 Windows 功能包括： + +- 由 containerd 提供支持的使用 RKE2 的 Windows 容器 +- 直接从 Rancher UI 配置 Windows RKE2 自定义集群 +- 用于 Windows RKE2 自定义集群的 Calico CNI +- 技术预览包含了 Windows Server 的 SAC 版本（2004 和 20H2） + +要使 Windows 支持 RKE2 自定义集群，请选择 Calico 作为 CNI。 + +:::note + +默认情况下，Rancher 允许 Windows 工作负载 pod 部署在 Windows 和 Linux Worker 节点上。在 RKE2 中创建混合集群时，你必须编辑 Chart 中的 `nodeSelector`，从而将 Pod 放置到兼容的 Windows 节点上。有关如何使用 `nodeSelector` 将 pod 分配给节点的更多信息，请参阅 [Kubernetes 文档](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#nodeselector)。 + +::: + +- Kubernetes v1.24.1 及更高版本支持 Windows RKE2 中的 HostProcess 容器。有关详细信息，请参阅[上游文档](https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/)。 + +## Windows 集群的要求 + +网络、操作系统和 Docker 的一般节点要求与 [Rancher 安装](../../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md) 的节点要求相同 + +### 操作系统和 Docker 要求 + +我们对 Windows Server 和 Windows 容器的支持与 LTSC（长期服务渠道）和 SAC（半年渠道）的 Microsoft 官方生命周期相匹配。 + +有关 Windows Server 的支持生命周期的日期，请参阅 [Microsoft 文档](https://docs.microsoft.com/en-us/windows-server/get-started/windows-server-release-info)。 + +### Kubernetes 版本 + +需要 Kubernetes v1.15+。 + +如果你在 Windows Server 20H2 Standard Core 上使用 Kubernetes v1.21，则必须在节点上安装补丁“2019-08 Servicing Stack Update for Windows Server”。 + +### 节点要求 + +集群中的主机至少需要： + +- 2 核 CPU +- 5 GB 内存 +- 50 GB 磁盘空间 + +Rancher 不会配置不满足要求的节点。 + +### 网络要求 + +在配置新集群之前，请确保你已经在接收入站网络流量的设备上安装了 Rancher。这是集群节点与 Rancher 通信所必需的。如果你尚未安装 Rancher，请在继续阅读本指南之前先参阅[安装文档](../../../../getting-started/installation-and-upgrade/installation-and-upgrade.md)进行安装。 + +Rancher 仅支持使用 Flannel 作为网络提供商的 Windows。 + +有两个网络选项：[**Host Gateway (L2bridge)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) 和 [**VXLAN (Overlay)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan)。 默认选项是 **VXLAN (Overlay)** 模式。 + +对于 **Host Gateway (L2bridge)** 网络，最好为所有节点使用相同的第 2 层网络。否则，你需要为它们配置路由规则。有关详细信息，请参阅[配置云托管 VM 路由的文档](network-requirements-for-host-gateway.md#云托管虚拟机的路由配置)。如果你使用的是 Amazon EC2、Google GCE 或 Azure 虚拟机，你需要[禁用私有 IP 地址检查](network-requirements-for-host-gateway.md#禁用私有-ip-地址检查)。 + +对于 **VXLAN (Overlay)** 网络, 必须安装 [KB4489899](https://support.microsoft.com/en-us/help/4489899) 补丁。大多数云托管虚拟机已经安装了此修复程序。 + +如果你在为 AWS 虚拟私有云配置 DHCP 选项集，请注意，你只能在 `domain-name` 选项字段中指定一个域名。详情请参见 [DHCP 选项文档](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_DHCP_Options.html)。 + +:::note + +一些 Linux 操作系统支持以空格分隔的多个域名。但是，其他 Linux 操作系统和 Windows 将该值视为单个域名，从而导致意外错误。如果你的 DHCP 选项集与具有多个操作系统实例的 VPC 相关联，请仅指定一个域名。 + +::: + +### 带有 ESXi 6.7u2 及更高版本的 vSphere 上的 Rancher + +如果你在带有 ESXi 6.7u2 或更高版本的 VMware vSphere 上使用 Rancher，并使用 Red Hat Enterprise Linux 8.3、CentOS 8.3 或 SUSE Enterprise Linux 15 SP2 或更高版本，你需要禁用 `vmxnet3` 虚拟网络适配器硬件卸载功能。否则，不同集群节点上的 pod 之间的所有网络连接会因为超时错误而失败。从 Windows pod 到在 Linux 节点上运行的关键服务（例如 CoreDNS）的所有连接也将失败。外部连接也可能失败。出现这个问题的原因是 Linux 发行版在 `vmxnet3` 中启用了硬件卸载功能，而且 `vmxnet3` 硬件卸载功能中存在一个会丢弃客户覆盖流量的数据包的 bug。要解决此问题，必须禁用 `vmxnet3` 硬件卸载功能。此设置不会在重启后继续生效，因此需要在每次启动时禁用。推荐的做法是在 `/etc/systemd/system/disable_hw_offloading.service` 中创建一个 systemd 单元文件，这会在启动时禁用 `vmxnet3` 硬件卸载功能。禁用 `vmxnet3` 硬件卸载功能的示例 systemd 单元文件如下所示。注意，`` 必须自定义为主机的 `vmxnet3` 网络接口，如 `ens192`： + +``` +[Unit] +Description=Disable vmxnet3 hardware offloading feature + +[Service] +Type=oneshot +ExecStart=ethtool -K tx-udp_tnl-segmentation off +ExecStart=ethtool -K tx-udp_tnl-csum-segmentation off +StandardOutput=journal + +[Install] +WantedBy=multi-user.target +``` + +然后在 systemd 单元文件上设置适当的权限： + +``` +chmod 0644 /etc/systemd/system/disable_hw_offloading.service +``` + +最后，启用 systemd 服务： + +``` +systemctl enable disable_hw_offloading.service +``` + +### 架构要求 + +Kubernetes 集群管理节点（`etcd` 和 `controlplane`）必须运行在 Linux 节点上。 + +部署工作负载的 `worker` 节点通常是 Windows 节点，但必须至少有一个 `worker` 节点运行在 Linux 上，才能按顺序运行 Rancher Cluster Agent、DNS、Metrics Server 和 Ingress 相关容器。 + +#### 推荐架构 + +我们推荐下表中列出的三节点架构，但你始终可以添加额外的 Linux 和 Windows worker 节点来扩展集群，从而实现冗余： + +| 节点 | 操作系统 | Kubernetes 集群角色 | 用途 | +| ------ | --------------------------------------------- | --------------------------- | ----------------------------------------------------------- | +| 节点 1 | Linux (推荐 Ubuntu Server 18.04) | Control plane, etcd, worker | 管理 Kubernetes 集群 | +| 节点 2 | Linux (推荐 Ubuntu Server 18.04) | Worker | 支持集群的 Rancher 集群代理、Metrics 服务器、DNS 和 Ingress | +| 节点 3 | Windows（Windows Server 核心版本 1809 或更高版本） | Worker | 运行 Windows 容器 | + +### 容器要求 + +Windows 要求容器的版本必须与部署容器的 Windows Server 的版本一致。因此，你必须在 Windows Server 核心版本 1809 或更高版本上构建容器。如果你已经使用早期的 Windows Server 核心版本构建了容器，则必须使用 Windows Server 核心版本 1809 或更高版本重新构建容器。 + +### 云提供商特定要求 + +如果你在集群中设置了 Kubernetes 云提供商，则需要进行一些额外的操作。如果你想使用云提供商的功能，例如为集群自动配置存储、负载均衡器或其他基础设施，你可能需要设置云提供商。有关如何配置满足条件的云提供商集群节点，请参阅[此页面](../set-up-cloud-providers/set-up-cloud-providers.md)。 + +如果你的云提供商是 GCE（Google Compute Engine），则必须执行以下操作： + +- 按照[步骤](../set-up-cloud-providers/google-compute-engine.md)在`cluster.yml` 中启用 GCE 云提供商。 +- 在 Rancher 中配置集群时，在 Rancher UI 中选择**自定义云提供商**作为云提供商。 + +## 教程： 如何创建支持 Windows 的集群 + +本教程描述了如何使用[推荐架构](#推荐架构)中的三个节点创建由 Rancher 配置的集群。 + +在在现有节点上使用 Rancher 配置集群时，你需要在每个节点上安装 [Rancher Agent](../../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/rancher-agent-options.md)来将节点添加到集群中。在 Rancher UI 中创建或编辑集群时，你会看到一个**自定义节点运行命令**，你可以在每台服务器上运行该命令，从而将服务器添加到集群中。 + +要设置支持 Windows 节点和容器的集群，你需要完成以下任务： + +### 1. 配置主机 + +要在具有 Windows 支持的现有节点上配置集群，请准备好你的主机。 + +主机可以是： + +- 云托管的虚拟机 +- 虚拟化集群中的虚拟机 +- 裸金属服务器 + +你将配置三个节点： + +- 一个 Linux 节点，用于管理 Kubernetes controlplane 并存储你的 `etcd`。 +- 第二个 Linux 节点，它将作为 worker 节点。 +- Windows 节点，它将作为 worker 节点运行 Windows 容器。 + +| 节点 | 操作系统 | +| ------ | --------------------------------------------- | +| 节点 1 | Linux (推荐 Ubuntu Server 18.04) | +| 节点 2 | Linux (推荐 Ubuntu Server 18.04) | +| 节点 3 | Windows（Windows Server 核心版本 1809 或更高版本） | + +如果你的节点托管在**云提供商**上，并且你需要自动化支持（例如负载均衡器或持久存储设备），你的节点还需要满足额外的配置要求。详情请参见[选择云提供商](../set-up-cloud-providers/set-up-cloud-providers.md)。 + +### 2. 在现有节点上创建集群 + +在现有节点上创建 Windows 集群的说明与一般[创建自定义集群的说明](../../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)非常相似，但有一些特定于 Windows 的要求。 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面上，单击**创建**。 +1. 单击**自定义**。 +1. 在**集群名称**字段中输入集群的名称。 +1. 在 **Kubernetes 版本**下拉菜单中，选择 v1.19 或更高版本。 +1. 在**网络提供商**字段中，选择 **Flannel**。 +1. 在 **Windows 支持**中，单击**启用**。 +1. 可选：启用 Windows 支持后，你将能够选择 Flannel 后端模式。有两个网络选项：[**Host Gateway (L2bridge)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#host-gw) 和 [**VXLAN (Overlay)**](https://github.com/coreos/flannel/blob/master/Documentation/backends.md#vxlan)。 默认选项是 **VXLAN (Overlay)** 模式。 +1. 点击**下一步**。 + +:::note 重要提示： + +对于 Host Gateway (L2bridge) 网络，最好为所有节点使用相同的第 2 层网络。否则，你需要为它们配置路由规则。有关详细信息，请参阅[配置云托管 VM 路由的文档](network-requirements-for-host-gateway.md#云托管虚拟机的路由配置)。如果你使用的是 Amazon EC2、Google GCE 或 Azure 虚拟机，你需要[禁用私有 IP 地址检查](network-requirements-for-host-gateway.md#禁用私有-ip-地址检查)。 + +::: + +### 3. 将节点添加到集群 + +本节介绍如何将 Linux 和 Worker 节点注册到集群。你将在每个节点上运行一个命令，该命令将安装 Rancher Agent 并允许 Rancher 管理每个节点。 + +#### 添加 Linux master 节点 + +在本节中，你需要在 Rancher UI 上填写表单以获取自定义命令，从而在 Linux master 节点上安装 Rancher Agent。然后，复制该命令并在 Linux master 节点上运行命令，从而在集群中注册该节点。 + +集群中的第一个节点应该是具有 **controlplane** 和 **etcd** 角色的 Linux 主机。至少必须为此节点启用这两个角色，并且必须先将此节点添加到集群中，然后才能添加 Windows 主机。 + +1. 在**节点操作系统**中，单击 **Linux**。 +1. 在**节点角色**中，至少选择 **etcd** 和 **controlplane**。推荐选择所有的三个角色。 +1. 可选：如果点击**显示高级选项**，你可以自定义 [Rancher Agent](../../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/rancher-agent-options.md)和[节点标签](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)的设置。 +1. 将屏幕上显示的命令复制到剪贴板。 +1. SSH 到你的 Linux 主机，然后运行复制到剪贴板的命令。 +1. 完成配置 Linux 节点后，选择**完成**。 + +**结果**： + +你已创建集群，集群的状态是**配置中**。Rancher 已在你的集群中。 + +当集群状态变为 **Active** 后，你可访问集群。 + +**Active** 状态的集群会分配到两个项目： + +- `Default`：包含 `default` 命名空间 +- `System`：包含 `cattle-system`，`ingress-nginx`，`kube-public` 和 `kube-system` 命名空间。 + + +节点可能需要几分钟才能注册到集群中。 + +#### 添加 Linux Worker 节点 + +在本节中，我们通过运行命令将 Linux Worker 节点注册到集群中。 + +在初始配置集群之后，你的集群只有一个 Linux 主机。接下来，我们添加另一个 Linux `worker` 主机，用于支持集群的 _Rancher Cluster Agent_、_Metrics Server_、_DNS_ 和 _Ingress_。 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 转到你创建的集群，然后单击 **⋮ > 编辑配置**。 +1. 向下滚动到**节点操作系统**。选择 **Linux**。 +1. 在**自定义节点运行命令**中，转到**节点选项**并选择 **Worker** 角色。 +1. 将屏幕上显示的命令复制到剪贴板。 +1. 使用远程终端连接登录到你的 Linux 主机。粘贴剪贴板的命令并运行。 +1. 在 **Rancher**中，单击**保存**。 + +**结果**：**Worker** 角色已安装在你的 Linux 主机上，并且节点会向 Rancher 注册。节点可能需要几分钟才能注册到集群中。 + +:::note + +Linux Worker 节点上的污点 + +以下污点将添加集群中的 Linux Worker 节点中。将此污点添加到 Linux Worker 节点后，添加到 Windows 集群的任何工作负载都将自动调度到 Windows Worker 节点。如果想将工作负载专门调度到 Linux Worker 节点上，则需要为这些工作负载添加容忍度。 + +| 污点键 | 污点值 | 污点效果 | +| -------------- | ---------- | ------------ | +| `cattle.io/os` | `linux` | `NoSchedule` | + +::: + +#### 添加 Windows Worker 节点 + +在本节中，我们通过运行命令将 Windows Worker 节点注册到集群中。 + +你可以通过编辑集群并选择 **Windows** 选项，从而将 Windows 主机添加到集群中。 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 转到你创建的集群，然后单击 **⋮ > 编辑配置**。 +1. 向下滚动到**节点操作系统**。选择 **Windows**。注意：你将看到 **worker** 角色是唯一可用的角色。 +1. 将屏幕上显示的命令复制到剪贴板。 +1. 使用你喜欢的工具（例如 [Microsoft 远程桌面](https://docs.microsoft.com/en-us/windows-server/remote/remote-desktop-services/clients/remote-desktop-clients)）登登录到 Windows 主机。在 **Command Prompt (CMD)** 中运行复制到剪贴板的命令。 +1. 在 Rancher 中，单击**保存**。 +1. 可选：如果要向集群添加更多 Windows 节点，请重复这些操作。 + +**结果**：**Worker** 角色已安装在你的 Windows 主机上，并且节点会向 Rancher 注册。节点可能需要几分钟才能注册到集群中。你现在已拥有一个 Windows Kubernetes 集群。 + +### 可选的后续步骤 + +创建集群后，你可以通过 Rancher UI 访问集群。最佳实践建议你设置以下访问集群的备用方式： + +- **通过 kubectl CLI 访问你的集群**：按照[这些步骤](../../manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#在工作站使用-kubectl-访问集群)在你的工作站上使用 kubectl 访问集群。在这种情况下，你将通过 Rancher Server 的身份验证代理进行身份验证，然后 Rancher 会让你连接到下游集群。此方法允许你在没有 Rancher UI 的情况下管理集群。 +- **通过 kubectl CLI 使用授权的集群端点访问你的集群**：按照[这些步骤](../../manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#直接使用下游集群进行身份验证)直接使用 kubectl 访问集群，而无需通过 Rancher Server 进行身份验证。我们建议设置此替代方法来访问集群，以便在无法连接到 Rancher 时访问集群。 + +## Azure 中存储类的配置 + +如果你的节点使用 Azure VM，则可以使用 [Azure 文件](https://docs.microsoft.com/en-us/azure/aks/azure-files-dynamic-pv)作为集群的存储类（StorageClass）。详情请参见[此部分](azure-storageclass-configuration.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md new file mode 100644 index 00000000000..b50aecb39a9 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md @@ -0,0 +1,30 @@ +--- +title: Horizontal Pod Autoscaler +description: 了解 Pod 水平自动扩缩 (HPA)。如何管理 HPA 以及如何使用服务部署来进行测试 +--- + + + + + +[Horizontal Pod Autoscaler（HPA）](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)是一项 Kubernetes 功能，用于将集群配置为自动扩缩其运行的服务。 + +Rancher 提供了一些附加功能来帮助管理 HPA，具体取决于 Rancher 的版本。 + +你可以使用 Rancher UI 创建、管理和删除 HPA。仅在 `autoscaling/v2beta2` API 中支持 HPA。 + +## 管理 HPA + +管理 HPA 的方式因你的 Kubernetes API 版本而异： + +- **Kubernetes API 版本 autoscaling/V2beta1**：允许根据应用程序的 CPU 和内存利用率自动扩缩 pod。 +- **Kubernetes API 版本 autoscaling/V2beta2**：允许根据 CPU 和内存利用率以及自定义指标自动扩缩 pod。 + +你可以使用 Rancher UI 创建、管理和删除 HPA。在 Rancher UI 中，你可以将 HPA 配置为根据 CPU 和内存利用率进行扩缩。有关详细信息，请参阅[使用 Rancher UI 管理 HPA](manage-hpas-with-ui.md)。如需根据自定义指标进行 HPA，你仍然需要使用 `kubectl`。有关详细信息，请参阅[配置 HPA 以使用 Prometheus 自定义指标进行扩缩](manage-hpas-with-kubectl.md#使用资源指标cpu-和内存配置-hpa-以进行扩缩)。 + +在 Rancher 2.0.7 及更高版本中创建的集群自动满足使用 HPA 的所有要求（metrics-server 和 Kubernetes 集群配置）。 +## 使用服务部署测试 HPA + +你可以转到你的项目并单击 **资源 > HPA** 来查看​​ HPA 当前的副本数。有关详细信息，请参阅[获取 HPA 指标和状态](manage-hpas-with-ui.md)。 + +你还可以使用 `kubectl` 来获取你使用负载测试工具测试的 HPA 的状态。有关详细信息，请参阅[使用 kubectl 测试 HPA](test-hpas-with-kubectl.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md new file mode 100644 index 00000000000..4dd86bb2a6c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md @@ -0,0 +1,67 @@ +--- +title: Kubernetes 资源 +--- + + + + + +你可以在 Rancher UI 中查看和操作 Kubernetes 集群中的所有自定义资源和 CRD。 + +## 工作负载 + +使用[工作负载](workloads-and-pods/workloads-and-pods.md)将应用部署到集群节点，工作负载是包含用于运行应用的 pod 的对象，以及为部署行为设置规则的元数据。工作负载可以部署在集群范围内，也可以部署在一个命名空间内。 + +部署工作负载时，你可以使用任何镜像进行部署。可供选择的[工作负载类型](workloads-and-pods/workloads-and-pods.md#工作负载类型)有多种，工作负载类型决定了你的应用程序的运行方式。 + +在工作负载部署之后，你可以继续使用它。你可以： + +- 将工作负载[升级](workloads-and-pods/upgrade-workloads.md)到它运行的应用的更新版本。 +- 如果升级出现问题，将工作负载[回滚](workloads-and-pods/roll-back-workloads.md)到以前的版本。 +- [添加一个 sidecar](workloads-and-pods/add-a-sidecar.md)，这是一个支持主要工作负载的工作负载。 + +## 负载均衡和 Ingress + +### 负载均衡器 + +启动应用程序后，它仅在集群中可用。无法从外部访问它。 + +如果你希望你的应用程序可以从外部访问，则必须向集群添加负载均衡器。如果用户知道负载均衡器的 IP 地址和应用的端口号，负载均衡器可以为外部连接创建一个访问集群的网关。 + +Rancher 支持两种类型的负载均衡器： + +- [Layer-4 负载均衡器](load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing.md#layer-4-load-balancer#四层负载均衡器) +- [Layer-7 负载均衡器](load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing.md#七层负载均衡器) + +有关详细信息，请参阅[负载均衡器](load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing.md)。 + +#### Ingress + +负载均衡器只能处理每个 service 的一个 IP 地址。换言之，如果你在集群中运行了多个 service，则必须为每个 service 配备一个负载均衡器。运行多个负载均衡器的花费可能非常高昂。因此，你可以使用 Ingress 来解决此问题。 + +Ingress 是一组充当负载均衡器的规则。Ingress 与一个或多个 Ingress Controller 一起动态路由 service 的请求。Ingress 收到请求时，集群中的 Ingress Controller 会对负载均衡器进行配置，从而根据你配置的 service 子域或路径规则将请求定向到正确的 service。 + +有关详细信息，请参阅 [Ingress](load-balancer-and-ingress-controller/add-ingresses.md)。 + +在项目中使用 Ingress 时，你可以通过设置全局 DNS 条目来将 Ingress 主机名编程到外部 DNS。 + +## 服务发现 + +使用负载均衡器和/或 Ingress 将集群公开给外部请求后，你只能通过 IP 地址访问集群。要创建可解析的主机名，你必须创建服务记录，该记录将 IP 地址、外部主机名、DNS 记录别名、工作负载或标记的 pod 映射到特定主机名。 + +有关详细信息，请参阅[服务发现](create-services.md)。 + +## 应用程序 + +除了启动应用程序的各个组件外，你还可以使用 Rancher 应用商店来启动应用，即 Helm Chart。 + +## Kubernetes 资源 + +在 Rancher 项目或命名空间的上下文中，_资源_ 是支持 Pod 操作的文件和数据。在 Rancher 中，证书、镜像仓库和密文都被视为资源。但是，Kubernetes 将资源划分为不同类型的[密文（secret）](https://kubernetes.io/docs/concepts/configuration/secret/)。因此，在单个项目或命名空间中，各个资源必须具有唯一的名称以避免冲突。资源主要用于承载敏感信息，但也有其他用途。 + +资源包括： + +- [证书](encrypt-http-communication.md)：用于加密/解密进入或离开集群的数据的文件。 +- [ConfigMap](configmaps.md)：存储一般配置信息的文件，例如一组配置文件。 +- [密文](secrets.md)：存储密码、token 或密钥等敏感数据的文件。 +- [镜像仓库](kubernetes-and-docker-registries.md)：携带用于验证私有镜像仓库的凭证的文件。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/ingress-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/ingress-configuration.md index 380629411b4..3ff047bc108 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/ingress-configuration.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/ingress-configuration.md @@ -9,7 +9,6 @@ description: Ingress 配置 由于 controller 的这一更改，默认 RKE1 配置不再将 `hostNetwork` 设置为 `true`。但是，你必须将 `hostNetwork` 设置为 `true` 才能使基于 TCP 和 UDP 的 Service 正常工作。为此，请[编辑](../../../../reference-guides/cluster-configuration/rancher-server-configuration/rke1-cluster-configuration.md#使用-yaml-编辑集群)集群的 YAML 并按照[官方 RKE1 文档](https://rke.docs.rancher.com/config-options/add-ons/ingress-controllers#configuring-network-options)中的步骤操作。 - ::: ## 指定要使用的主机名 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md new file mode 100644 index 00000000000..00ae5a04894 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md @@ -0,0 +1,65 @@ +--- +title: 在 Rancher 中设置负载均衡器和 Ingress Controller +description: 了解如何设置负载均衡器和 Ingress Controller 以在 Rancher 中重定向服务请求，并了解负载均衡器的限制 +--- + + + + + +在 Rancher 中，你可以通过设置负载均衡器和 Ingress Controller 来重定向服务请求。 + +## 负载均衡器 + +启动应用程序后，该应用程序仅在集群内可用。你无法从集群外部访问它。 + +如果你希望从外部访问应用程序，则必须向集群添加负载均衡器或 Ingress。如果用户知道负载均衡器的 IP 地址和应用的端口号，负载均衡器可以为外部连接创建一个访问集群的网关。 + +Rancher 支持两种类型的负载均衡器： + +- [Layer-4 负载均衡器](layer-4-and-layer-7-load-balancing.md#四层负载均衡器) +- [Layer-7 负载均衡器](layer-4-and-layer-7-load-balancing.md#七层负载均衡器) + +有关详细信息，请参阅[负载均衡器](layer-4-and-layer-7-load-balancing.md)。 + +### 负载均衡器限制 + +负载均衡器有几个需要注意的限制： + +- 负载均衡器只能处理每个 service 的一个 IP 地址。换言之，如果你在集群中运行了多个 service，则必须为每个 service 配备一个负载均衡器。运行多个负载均衡器的花费可能非常高昂。 + +- 如果你想将负载均衡器与托管的 Kubernetes 集群（即托管在 GKE、EKS 或 AKS 中的集群）一起使用，则负载均衡器必须运行在该云提供商的基础设施上。请根据你配置集群的方式查看负载均衡器的兼容列表： + +- [支持 Layer-4 负载均衡](layer-4-and-layer-7-load-balancing.md#四层负载均衡支持) + +- [支持 Layer-7 负载均衡](layer-4-and-layer-7-load-balancing.md#七层负载均衡支持) + +## Ingress + +如上所述，使用负载均衡器的缺点是： + +- 每个服务负载均衡器只能处理一个 IP 地址。 +- 如果你在集群中运行多个服务，则每个服务都必须配备一个负载均衡器。 +- 为每个服务配备负载均衡器的花费可能非常高昂。 + +相反，如果将某个 Ingress 用作集群的入口点，Ingress 可以更灵活地将流量路由到多个 service。它可以将多个 HTTP 请求映射到 service，而无需为每个 service 提供单独的 IP 地址。 + +因此，如果你需要使用相同的 IP 地址、Layer 7 协议或特权节点端口（80 和 443）来公开多个 service，你可以使用一个 Ingress。 + +Ingress 与一个或多个 Ingress Controller 一起动态路由 service 的请求。Ingress 收到请求时，集群中的 Ingress Controller 会根据你配置的 service 子域或路径规则将请求定向到正确的 service。 + +每个 Kubernetes Ingress 资源都对应一个 `/etc/nginx/sites-available/` 中的文件，其中包含一个配置对特定文件和文件夹的请求的 `server{}` 配置块。 + +Ingress 能为你的集群创建一个入口端口（与负载均衡器类似），可以位于集群的内部或外部。RKE 启动的集群中的 Ingress 和 Ingress Controller 由 [Nginx](https://www.nginx.com/) 提供支持。 + +Ingress 还支持其他功能，例如 SSL 终止、基于名称的虚拟主机等。 + +:::note 在高可用性配置中使用 Rancher： + +请避免将 Ingress 添加到 `local` 集群。Rancher 将 Nginx Ingress Controller 作为 Rancher 管理的 _所有_ 集群的全局入口点，其中包括 `local` 集群。因此，当用户尝试访问应用程序时，Rancher 可能会由于重新加载 Nginx 配置而断开连接。要解决这个问题，我们建议你仅在通过 Rancher 启动的集群中部署应用程序。 + +::: + +- 有关如何在 Rancher 中设置 Ingress 的更多信息，请参阅 [Ingress](add-ingresses.md)。 +- 有关 Ingress 和 Ingress Controller 的完整信息，请参阅 [Kubernetes Ingress 文档](https://kubernetes.io/docs/concepts/services-networking/ingress/)。 +- 在项目中使用 Ingress 时，你可以通过设置全局 DNS 条目来将 Ingress 主机名编程到外部 DNS。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/deploy-workloads.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/deploy-workloads.md index 16ae92f17ee..6fecebc7341 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/deploy-workloads.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/deploy-workloads.md @@ -41,7 +41,7 @@ description: 阅读此步骤指南以部署工作负载。部署工作负载以 - 在 [AWS](https://aws.amazon.com/) 中，节点必须位于同一可用区中并具有附加/分离卷的 IAM 权限。 - - 集群必须使用 [AWS 云提供商](https://github.com/kubernetes/website/blob/release-1.18/content/en/docs/concepts/cluster-administration/cloud-providers.md#aws)选项。有关启用此选项的更多信息，请参阅[创建 AWS EC2 集群](../../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-amazon-ec2-cluster.md)或[创建自定义集群](../../../../pages-for-subheaders/use-existing-nodes.md)。 + - 集群必须使用 [AWS 云提供商](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#aws)选项。有关启用此选项的更多信息，请参阅[创建 AWS EC2 集群](../../launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-amazon-ec2-cluster.md)或[创建自定义集群](../../../../pages-for-subheaders/use-existing-nodes.md)。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md new file mode 100644 index 00000000000..08be0859bb8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md @@ -0,0 +1,82 @@ +--- +title: "Kubernetes 工作负载和 Pod" +description: "了解在 Kubernetes 中构建复杂容器化应用程序的两种结构：Kubernetes 工作负载和 Pod" +--- + + + + + +你可以使用两种基本结构（pod 和工作负载）在 Kubernetes 中构建复杂的容器化应用程序。构建应用程序后，你可以使用第三种结构（service）在集群中或互联网上公开应用程序。 + +### Pod + +[_Pod_](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/) 是一个或多个共享网络命名空间和存储卷的容器。大多数 pod 只有一个容器。因此，我们讨论的 _pod_ 通常等同于 _容器_。扩展 pod 的方式与扩展容器的方式相同，即配置实现服务的同一 pod 的多个实例。通常，Pod 会根据工作负载进行扩展和管理。 + +### 工作负载 + +_工作负载_ 是为 pod 设置部署规则的对象。Kubernetes 基于这些规则执行部署，并根据应用程序的当前状态来更新工作负载。 +工作负载让你可以定义应用程序调度、扩展和升级的规则。 + +#### 工作负载类型 + +Kubernetes 将工作负载分为不同的类型。Kubernetes 支持的最流行的类型是： + +- [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) + + _Deployment_ 最适合用于无状态应用程序（即不需要维护工作负载的状态）。由 Deployment 类型工作负载管理的 Pod 是独立且一次性的。如果 pod 中断了，Kubernetes 会删除该 pod 然后重新创建它。一个示例应用程序是 Nginx Web 服务器。 + +- [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) + + 与 Deployment 相比，_StatefulSet_ 最适合在需要维护身份和存储数据的应用程序中使用。适用的应用程序类似于 Zookeeper（一个需要数据库进行存储的应用程序）。 + +- [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) + + _Daemonset_ 确保集群中的每个节点都运行 pod 的副本。如果你需要收集日志或监控节点性能，这种类似 daemon 的工作负载效果是最好的。 + +- [Job](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) + + _Job_ 启动一个或多个 Pod 并确保指定数量的 Pod 能成功终止。Job 最好用于运行有限任务至完成状态，而不是管理正在进行的应用程序的所需状态。 + +- [CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) + + _CronJobs_ 与 Job 类似。但是，CronJob 会基于 cron 的计划运行到完成状态。 + +### Services + +在许多用例中，工作负载必须： + +- 由集群中的其他工作负载访问。 +- 暴露给外部。 + +你可以通过创建一个 _Service_ 实现这些目的。Service 使用[选择器/标签（查看代码示例）](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#service-and-replicationcontroller)来映射到底层工作负载的 pod。Rancher UI 使用你选择的服务端口和类型来自动创建 service 以及工作负载，从而简化此映射过程。 + +#### Service 类型 + +Rancher 中有几种可用的 Service 类型。以下描述来自 [Kubernetes 文档](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types)。 + +- **ClusterIP** + + > 在集群内部 IP 上公开 Service。如果你选择此值，Service 只能从集群内访问。这是默认的 `ServiceType`。 + +- **NodePort** + + > 在每个节点 IP 上的静态端口（`NodePort`）上暴露 Service。`ClusterIP` service 是自动创建的，而 `NodePort` service 会路由到 ClusterIP service。你可以通过请求 `:` 在集群外部联系 `NodePort` service。 + +- **LoadBalancer** + + > 使用云提供商的负载均衡器向外部公开服务。`NodePort` 和 `ClusterIP` service 是自动创建的，外部负载均衡器会路由到这些 service。 + +## 工作负载选项 + +以下文档介绍了如何部署工作负载和使用工作负载选项。 + +- [部署工作负载](deploy-workloads.md) +- [升级工作负载](upgrade-workloads.md) +- [回滚工作负载](roll-back-workloads.md) + +## 相关链接 + +### 外部链接 + +- [Service](https://kubernetes.io/docs/concepts/services-networking/service/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md new file mode 100644 index 00000000000..cc4e205c941 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md @@ -0,0 +1,84 @@ +--- +title: 使用 Rancher 启动 Kubernetes +--- + + + + + +Rancher 可以使用任意节点启动 Kubernetes 集群。在 Rancher 中将 Kubernetes 部署到这些节点上时，你可以选择 [Rancher Kubernetes Engine](https://rancher.com/docs/rke/latest/en/) (RKE) 或 [RKE2](https://docs.rke2.io) 发行版。Rancher 可以在任何计算机上启动 Kubernetes，包括： + +- 裸金属服务器 +- 本地虚拟机 +- 由云厂商托管的虚拟机 + +Rancher 可以在现有节点上安装 Kubernetes，也可以在云厂商中动态配置节点并安装 Kubernetes。 + +Rancher 还可以创建节点池。在托管在云厂商的节点池上安装 Kubernetes 的一个好处是，如果一个节点与集群断开连接，Rancher 可以自动创建另一个节点并将其加入集群，从而确保节点池的数量符合要求。 + +## RKE + +### 要求 + +如果你使用 RKE 建立集群，节点必须满足下游集群的[节点要求](../kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)。 + +### 在云厂商的新节点上启动 Kubernetes + +使用 Rancher，你可以基于[节点模板](use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md#节点模板)创建节点池。此节点模板定义了要用于在云厂商中启动节点的参数。 + +在托管在云厂商的节点池上安装 Kubernetes 的一个好处是，如果一个节点与集群断开连接，Rancher 可以自动创建另一个节点并将其加入集群，从而确保节点池的数量符合要求。 + +有关详细信息，请参阅[在新节点上启动 Kubernetes](use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)。 + +### 在现有自定义节点上启动 Kubernetes + +在这种情况下，你希望将 Kubernetes 安装到裸机服务器、本地虚拟机或云厂商中已存在的虚拟机上。使用此选项，你将在主机上运行 Rancher Agent Docker 容器。 + +如果要重复使用之前的自定义集群中的节点，请在复用之前[清理节点](../manage-clusters/clean-cluster-nodes.md)。如果你重复使用尚未清理的节点，则集群配置可能会失败。 + +有关详细信息，请参阅[自定义节点](../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)。 + +### 以编程方式创建 RKE 集群 + +通过 Rancher 以编程方式部署 RKE 集群的最常见方法是使用 Rancher 2 Terraform Provider。详情请参见[使用 Terraform 创建集群](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster)。 + +## RKE2 + +Rancher 2.6 支持直接使用 Rancher UI 配置 [RKE2](https://docs.rke2.io/) 集群。RKE2，也称为 RKE Government，是一个完全符合标准的 Kubernetes 发行版，它专注于安全性和合规性。在 Rancher 2.6.4 及更早版本中，RKE2 配置还处于技术预览阶段。 + +在 Rancher 2.6.5 中，RKE2 已经 GA。 + +### 要求 + +如果你使用 RKE2 建立集群，节点必须满足下游集群的[节点要求](https://docs.rke2.io/install/requirements)。 + +### 在云厂商的新节点上启动 Kubernetes + +RKE2 基于使用上游[集群 API](https://github.com/kubernetes-sigs/cluster-api) 项目的新配置框架。这个新配置框架支持： + +- 将 RKE2 集群配置到 Rancher 具有主机驱动的任何提供商上 +- 完全在 Rancher 中配置 RKE2 集群 +- 除了 Canal 之外，还可以选择 CNI 选项， Calico、Cilium 和 Multus + +RKE2 配置还包括在具有 Windows 节点的集群上安装 RKE2。 + +RKE2 的 Windows 功能包括： + +- Windows 支持 vSphere 主机驱动 +- 用于 Windows RKE2 自定义集群的 Calico CNI +- Calico 的项目网络隔离 (PNI) +- 由 containerd 提供支持的使用 RKE2 的 Windows 容器 +- 通过 Terraform 配置 Windows RKE2 集群 +- 直接从 Rancher UI 配置 Windows RKE2 自定义集群 + +要使 Windows 支持 RKE2 自定义集群，请选择 Calico 作为 CNI。 + +### 在现有自定义节点上启动 Kubernetes + +RKE2 还支持在预配置的虚拟机或裸机节点上安装自定义集群。 + +如果要重复使用之前的自定义集群中的节点，请在复用之前清理节点。如果你重复使用尚未清理的节点，则集群配置可能会失败。 + +### 以编程方式创建 RKE2 集群 + +通过 Rancher 以编程方式部署 RKE2 集群的最常见方法是使用 Rancher 2 Terraform Provider。详情请参见[使用 Terraform 创建集群](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster_v2)。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md new file mode 100644 index 00000000000..885bbb38847 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md @@ -0,0 +1,21 @@ +--- +title: 创建 Nutanix AOS 集群 +description: 使用 Rancher 创建 Nutanix AOS (AHV) 集群。集群可能包括具有不同属性的 VM 组，这些属性可用于细粒度控制节点的大小。 +--- + + + + + +[Nutanix Acropolis 操作系统](https://www.nutanix.com/products/acropolis) (Nutanix AOS) 是适用于 Nutanix 超融合基础架构平台的操作系统。AOS 带有一个名为 [Acropolis Hypervisor（AHV）](https://www.nutanix.com/products/ahv)的内置虚拟机监控程序。你可以结合使用 Rancher 与 Nutanix AOS (AHV)，从而在本地体验云环境的操作。 + +Rancher 可以在 AOS (AHV) 中配置节点并在其上安装 Kubernetes。在 AOS 中创建 Kubernetes 集群时，Rancher 首先与 Prism Central API 通信来配置指定数量的虚拟机。然后在虚拟机上安装 Kubernetes。 + +Nutanix 集群可能由多组具有不同属性（例如内存或 vCPU 数量）的 VM 组成。这种分组允许对每个 Kubernetes 角色的节点大小进行细粒度控制。 + +- [创建 Nutanix 集群](provision-kubernetes-clusters-in-aos.md#creating-a-nutanix-aos-cluster) +- [配置存储](provision-kubernetes-clusters-in-aos.md) + +## 创建 Nutanix 集群 + +在[本节](provision-kubernetes-clusters-in-aos.md)中，你将学习如何使用 Rancher 在 Nutanix AOS 中安装 [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes 集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md new file mode 100644 index 00000000000..5bc6d65b953 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md @@ -0,0 +1,156 @@ +--- +title: 在云厂商的新节点上启动 Kubernetes +--- + + + + + +在 Rancher 中使用节点模板来创建 RKE 或 RKE2 集群时，每个生成的节点池都会显示在新的**主机池**选项卡中。你可以通过执行以下操作来查看主机池： + +1. 点击**☰ > 集群管理**。 +1. 单击 RKE 或 RKE2 集群的名称。 + +## RKE 集群 + +使用 Rancher，你可以基于[节点模板](use-new-nodes-in-an-infra-provider.md#节点模板)创建节点池。此节点模板定义了要用于在基础设施提供商或云厂商中启动节点的参数。 + +在托管在云厂商的节点池上安装 Kubernetes 的一个好处是，如果一个节点与集群断开连接，Rancher 可以自动创建另一个节点并将其加入集群，从而确保节点池的数量符合要求。 + +可用于创建节点模板的云提供商是由[主机驱动](use-new-nodes-in-an-infra-provider.md#主机驱动)决定的。 + +### 节点模板 + +节点模板保存了用于在特定云提供商中配置节点时要使用的参数。这些节点可以从 UI 启动。Rancher 使用 [Docker Machine](https://docs.docker.com/machine/) 来配置这些节点。可用于创建节点模板的云提供商取决于 Rancher 中状态是 Active 的主机驱动。 + +在 Rancher 中创建节点模板后，模板会被保存，以便你可以再次使用该模板来创建节点池。节点模板绑定到你的登录名。添加模板后，你可以将其从用户配置文件中删除。 + +#### 节点标签 + +你可以为每个节点模板添加[标签](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)，这样，使用节点模板创建的节点都会自动带有这些标签。 + +无效标签会阻止升级，或阻止 Rancher 启动。有关标签语法的详细信息，请参阅 [Kubernetes 文档](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set)。 + +#### 节点污点 + +你可以为每个节点模板添加[污点](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)，这样，使用节点模板创建的节点都会自动带有这些污点。 + +由于污点可以同时添加到节点模板和节点池中，因此如果添加了相同键的污点效果没有冲突，则所有污点都将添加到节点中。如果存在具有相同键但不同效果的污点，则节点池中的污点将覆盖节点模板中的污点。 + +#### 节点模板的管理员控制 + +管理员可以控制所有节点模板。现在，管理员可以维护 Rancher 中的所有节点模板。当节点模板所有者不再使用 Rancher 时，他们创建的节点模板可以由管理员管理，以便继续更新和维护集群。 + +要访问所有节点模板，管理员需要执行以下操作： + +1. 点击 **☰ > 集群管理**。 +1. 单击 **RKE1 配置 > 节点模板**。 + +**结果**：列出所有节点模板。你可以通过单击 **⋮** 来编辑或克隆模板。 + +### 节点池 + +使用 Rancher，你可以基于[节点模板](#节点模板)创建节点池。 + +节点模板定义了节点的配置，例如要使用的操作系统、CPU 数量和内存量。 + +使用节点池的好处是，如果一个节点被销毁或删除，你可以增加 Active 节点的数量来补偿丢失的节点。节点池可以帮助你确保节点池的计数符合要求。 + +每个节点池必须分配一个或多个节点角色。 + +每个节点角色（即 etcd、controlplane 和 worker）都应分配给不同的节点池。虽然你可以将多个节点角色分配给同一个节点池，但不要在生产集群中执行此操作。 + +推荐的设置： + +- 具有 etcd 角色且计数为 3 的节点池 +- 具有 controlplane 角色且计数至少为 2 的节点池 +- 具有 worker 角色且计数至少为 2 的节点池 + +**离线环境中的 RKE1 下游集群节点**： + +默认情况下，在配置 RKE1 下游集群节点时（例如在 vSphere 中），Rancher 会尝试运行 Docker 安装脚本。但是，Rancher Docker 安装脚本在离线环境中会运行失败。要解决此问题，如果 Docker 已预安装到 VM 镜像上，你可以选择在创建节点模板时跳过安装 Docker。为此，你可以在 Rancher UI **引擎选项**下的 `Docker 安装 URL` 下拉列表中选择 **无**。 + +
**引擎选项下拉列表**
+ +![引擎选项下拉列表](/img/node-template-engine-options-rke1.png) + +#### 节点池污点 + +如果你没有在节点模板上定义[污点](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)，则可以为每个节点池添加污点。将污点添加到节点池的好处是你可以更改节点模板，而不需要先确保污点存在于新模板中。 + +每个污点都将自动添加到节点池中已创建的节点。因此，如果你在已有节点的节点池中添加污点，污点不会应用到已有的节点，但是添加到该节点池中的新节点都将获得该污点。 + +如果污点同时添加到节点模板和节点池中，且添加了相同键的污点效果没有冲突，则所有污点都将添加到节点中。如果存在具有相同键但不同效果的污点，则节点池中的污点将覆盖节点模板中的污点。 + +#### 节点自动替换 + +Rancher 可以自动替换节点池中无法访问的节点。如果节点在指定的时间中处于 Inactive 状态，Rancher 将使用该节点池的节点模板来重新创建节点。 + +:::caution + +自我修复节点池的功能帮助你替换无状态应用的 worker 节点。不建议在 master 节点或连接了持久卷的节点的节点池上启用节点自动替换，因为虚拟机会被临时处理。节点池中的节点与集群断开连接时，其持久卷将被破坏，从而导致有状态应用的数据丢失。 + +::: + +节点自动替换基于 Kubernetes 节点控制器工作。节点控制器定期检查所有节点的状态（可通过 `kube-controller` 的 `--node-monitor-period` 标志配置）。一个节点不可访问时，节点控制器将污染该节点。发生这种情况时，Rancher 将开始其删除倒计时。你可以配置 Rancher 等待删除节点的时间。如果在删除倒计时结束前污点没有被删除，Rancher 将继续删除该节点。Rancher 会根据节点池设置的数量来创建新的节点。 + +#### 启用节点自动替换 + +创建节点池时，你可以指定 Rancher 替换无响应节点的等待时间（以分钟为单位）。 + +1. 在创建或编辑集群的表单中，转到**节点池**。 +1. 转到要启用节点自动替换的节点池。在 **Recreate Unreachable After** 字段中，输入 Rancher 在替换节点之前应该等待节点响应的分钟数。 +1. 填写表单的其余部分以创建或编辑集群。 + +**结果** ：已为节点池启用节点自动替换。 + +#### 禁用节点自动替换 + +你可以执行以下步骤从 Rancher UI 禁用节点自动替换： + +1. 点击 **☰ > 集群管理**。 +1. 在**集群**页面上，转到要禁用节点自动替换的集群，然后单击 **⋮ > 编辑配置**。 +1. 在**节点池**部分中，转到要启用节点自动替换的节点池。在 **Recreate Unreachable After** 字段中，输入 0。 +1. 单击**保存**。 + +**结果**：已禁用节点池的节点自动替换。 + +### 云凭证 + +节点模板可以使用云凭证，来存储用于在云提供商中启动节点的凭证，其优点是： + +- 凭证会存储为更安全的 Kubernetes 密文，而且你无需每次都输入凭证便可编辑节点模板。 + +- 创建云凭证后，你可以重新使用该凭证来创建其他节点模板。 + +- 多个节点模板可以使用相同的云凭证来创建节点池。如果你的密钥被泄露或过期，则可以在一个位置更新云凭证，从而一次更新所有使用该凭证的节点模板。 + +创建云凭证后，用户可以[管理创建的云凭证](../../../../reference-guides/user-settings/manage-cloud-credentials.md)。 + +### 主机驱动 + +如果你找不到想要的主机驱动，你可以在 Rancher 的[内置主机驱动](../../authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md#激活停用主机驱动)中查看并激活它，也可以[添加自定义主机驱动](../../authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md#添加自定义主机驱动)。 + +## RKE2 集群 + +Rancher 2.6 支持直接使用 Rancher UI 配置 [RKE2](https://docs.rke2.io/) 集群。RKE2，也称为 RKE Government，是一个完全符合标准的 Kubernetes 发行版，它专注于安全性和合规性。 + +:::note + +对于 RKE2 集群模板，请参阅[此页面](../../manage-clusters/manage-cluster-templates.md#rke2-集群模板)了解更多信息。 + +::: + +### 节点角色 + +RKE2 CLI 公开了 `server` 和 `agent` 两个角色，它们分别代表 Kubernetes 节点角色 `etcd` + `controlplane` 和 `worker`。通过 Rancher 2.6 中的 RKE2 集成，RKE2 节点池可以分配更细粒度的角色，例如 `etcd` 和 `controlplane`。 + +你可以在 RKE2 CLI 中使用标志和节点污染，来控制调度工作负载和 Kubernetes master 节点的位置，从而使用 `etcd`，`controlplane` 和 `worker` 节点功能。这些角色没有在 RKE2 CLI 中实现为第一级角色的原因是，RKE2 被概念化为一组原始构建块，使用 Rancher 等编排系统得到最佳利用。 + +在 Rancher 中实现这三个节点角色，表示 Rancher 管理的 RKE2 集群能够轻松使用为 RKE 集群推荐的相同架构的所有最佳实践。 + +在[推荐的集群架构](../../kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/recommended-cluster-architecture.md)中，我们概述了每个角色集群应该有多少节点： + +- 至少拥有三个角色为 etcd 的节点，来确保失去一个节点时仍能存活。 +- 至少两个节点具有 controlplane 角色，以实现主组件高可用性。 +- 至少两个具有 worker 角色的节点，用于在节点故障时重新安排工作负载。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md index 25e4cf7e087..806090bff7c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md @@ -48,8 +48,6 @@ title: 在 vSphere 中配置 Kubernetes 集群 ## 创建 vSphere 集群 -在 Rancher 中创建 vSphere 集群的操作取决于 Rancher 的版本。 - 1. [创建云凭证](#1-创建云凭证) 2. [使用云凭证创建节点模板](#2-使用云凭证创建节点模板) 3. [使用节点模板创建具有节点池的集群](#3-使用节点模板创建具有节点池的集群) @@ -109,4 +107,4 @@ title: 在 vSphere 中配置 Kubernetes 集群 - **通过 kubectl CLI 访问你的集群**：按照[这些步骤](../../../../new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#在工作站使用-kubectl-访问集群)在你的工作站上使用 kubectl 访问集群。在这种情况下，你将通过 Rancher Server 的身份验证代理进行身份验证，然后 Rancher 会让你连接到下游集群。此方法允许你在没有 Rancher UI 的情况下管理集群。 - **通过 kubectl CLI 使用授权的集群端点访问你的集群**：按照[这些步骤](../../../../new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#直接使用下游集群进行身份验证)直接使用 kubectl 访问集群，而无需通过 Rancher 进行身份验证。我们建议设置此替代方法来访问集群，以便在无法连接到 Rancher 时访问集群。 -- **配置存储**：有关如何使用 Rancher 在 vSphere 中配置存储的示例，请参阅[本节](../../../../../pages-for-subheaders/provisioning-storage-examples.md)。要在 vSphere 中动态配置存储，你必须启用 vSphere 云提供商。有关更多信息，请参阅[树内 vSphere cloud provider 文档](../../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md)和[树外 vSphere cloud provider 文档](../../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 \ No newline at end of file +- **配置存储**：有关如何使用 Rancher 在 vSphere 中配置存储的示例，请参阅[本节](../../../../../pages-for-subheaders/provisioning-storage-examples.md)。要在 vSphere 中动态配置存储，你必须启用 vSphere 云提供商。有关更多信息，请参阅[树内 vSphere cloud provider 文档](../../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md)和[树外 vSphere cloud provider 文档](../../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md new file mode 100644 index 00000000000..396f6370108 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md @@ -0,0 +1,60 @@ +--- +title: 创建 vSphere 集群 +description: 使用 Rancher 创建 vSphere 集群。集群可能包括具有不同属性的 VM 组，这些属性可用于细粒度控制节点的大小。 +--- + + + + + +import YouTube from '@site/src/components/YouTube' + +你可以结合使用 Rancher 与 vSphere，从而在本地体验云环境的操作。 + +Rancher 可以在 vSphere 中配置节点并在其上安装 Kubernetes。在 vSphere 中创建 Kubernetes 集群时，Rancher 首先与 vCenter API 通信来配置指定数量的虚拟机。然后在它们之上安装 Kubernetes。 + +vSphere 集群可能由多组具有不同属性（例如内存或 vCPU 数量）的 VM 组成。这种分组允许对每个 Kubernetes 角色的节点大小进行细粒度控制。 + +## Rancher v2.3 中的 vSphere 增强功能 + +我们更新了 vSphere 节点模板，使你可以通过以下增强功能在本地体验云操作： + +### 自我修复的节点池 + +使用 Rancher 配置 vSphere 节点的最大优势之一，是允许你在本地集群中使用 Rancher 的自我修复节点池（也称为[节点自动替换功能](../use-new-nodes-in-an-infra-provider.md#节点自动替换)）。自我修复节点池的功能帮助你替换无状态应用的 worker 节点。当 Rancher 使用节点模板配置节点时，Rancher 可以自动替换无法访问的节点。 + +:::caution + +不建议在 master 节点或连接了持久卷的节点的节点池上启用节点自动替换，因为虚拟机会被临时处理。节点池中的节点与集群断开连接时，其持久卷将被破坏，从而导致有状态应用的数据丢失。 + +::: + +### 实例和调度的动态填充选项 + +vSphere 的节点模板已更新。当你使用 vSphere 凭证创建节点模板时，该模板会自动填充你在 vSphere 控制台中可以访问的相同的虚拟机配置选项。 + +要填充的字段设置需要满足[先决条件](provision-kubernetes-clusters-in-vsphere.md#vsphere-中的准备工作)。 + +### 更多支持的操作系统 + +你可以使用任何支持 `cloud-init` 的操作系统来配置 VM。[cloud config](https://cloudinit.readthedocs.io/en/latest/topics/examples.html) 仅支持 YAML 格式。 + +### v2.3.3 节点模板功能的视频介绍 + +在这段 YouTube 视频中，我们演示了如何使用新的节点模板在本地环境体验云环境一样的操作。 + + + +## 创建 vSphere 集群 + +在[本节](provision-kubernetes-clusters-in-vsphere.md)中，你将学习如何使用 Rancher 在 vSphere 中安装 [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes 集群。 + +## 配置存储 + +有关如何使用 Rancher 在 vSphere 中配置存储的示例，请参阅[本节](../../../manage-clusters/provisioning-storage-examples/vsphere-storage.md)。要在 vSphere 中动态配置存储，你必须启用 vSphere 云提供商。请参阅[树内 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md)和[树外 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 + +## 启用 vSphere 云提供商 + +在 Rancher 中设置云提供商时，Rancher Server 可以自动为集群配置新的基础设施，包括新节点或持久存储设备。 + +有关启用 vSphere 云提供商的详细信息，请参阅[树内 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md)和[树外 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md new file mode 100644 index 00000000000..e0d0dc17cd8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/access-clusters.md @@ -0,0 +1,65 @@ +--- +title: 集群访问 +--- + + + + + +本节介绍可以用来访问 Rancher 管理的集群的工具。 + +有关如何授予用户访问集群的权限的信息，请参阅[将用户添加到集群](add-users-to-clusters.md)部分。 + +有关 RBAC 的更多信息，请参阅[本节](../../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md)。 + +有关如何设置认证的信息，请参阅[本节](../../authentication-permissions-and-global-configuration/authentication-config/authentication-config.md)。 + +## Rancher UI 中的集群 + +我们提供了多种通过 Rancher UI 查看和管理集群的方法。 + +### 集群页面 + +从 **☰** 菜单访问**集群**页面： + +1. 单击 **☰**。 +1. 选择**集群管理**。 + +你还可以通过单击 Rancher UI **主页**集群表格上方的**管理**按钮来访问**集群**页面。 + +在**集群**页面上，选择每行末尾的 **⁝** 以查看包含以下选项的子菜单： + +* [Kubectl Shell](use-kubectl-and-kubeconfig.md) +* 下载 KubeConfig +* 将 KubeConfig 复制到剪切板 +* 编辑配置 +* 查看 YAML +* 下载 YAML + +### 集群仪表板 + +在**集群**页面上，选择每行末尾的**浏览**按钮查看该集群的**集群仪表板**。你还可以通过单击表中集群的名称，然后单击**集群**页面上的**浏览**按钮来查看仪表板。 + +也可以通过单击集群名称从 Rancher UI **主页**访问**集群仪表板**。 + +你还可以从顶部导航栏中的 **☰** 访问**集群仪表板**： + +1. 单击 **☰**。 +1. 从**浏览集群**菜单中选择集群的名称。 + +**集群仪表板**列出了集群相关的信息，例如节点数量、内存使用情况、事件和资源。 + +## kubectl + +你可以使用 Kubernetes 命令行工具 [kubectl](https://kubernetes.io/docs/reference/kubectl/overview/)，来管理你的集群。使用 kubectl 有两种选择： + +- **Rancher kubectl shell**：通过启动 Rancher UI 中可用的 kubectl shell 与集群交互。此选项不需要你进行任何配置操作。有关详细信息，请参阅[使用 kubectl Shell 访问集群](use-kubectl-and-kubeconfig.md)。 +- **终端远程连接**：你也可以通过在本地桌面上安装 [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)，然后将集群的 kubeconfig 文件复制到本地 `~/.kube/config` 目录来与集群交互。有关更多信息，请参阅[使用 kubectl 和 kubeconfig 文件访问集群](use-kubectl-and-kubeconfig.md)。 + +## Rancher CLI + +你可以下载 Rancher 自己的命令行工具 [Rancher CLI](../../../../reference-guides/cli-with-rancher/cli-with-rancher.md) 来控制你的集群。这个 CLI 工具可以直接与不同的集群和项目进行交互，或者向它们传递 `kubectl` 命令。 + +## Rancher API + +最后，你可以通过 Rancher API 与集群进行交互。在使用 API 之前，你必须先获取 [API 密钥](../../../../reference-guides/user-settings/api-keys.md)。要查看 API 对象的不同资源字段和操作，请打开 API UI（API UI 可以通过单击 Rancher UI 对象的**在 API 中查看**访问）。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/authorized-cluster-endpoint.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/authorized-cluster-endpoint.md index dee43a45458..364a4012eee 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/authorized-cluster-endpoint.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/authorized-cluster-endpoint.md @@ -4,11 +4,18 @@ title: 授权集群端点的工作原理 本文介绍 kubectl CLI、kubeconfig 文件和授权集群端点如何协同工作，使你可以直接访问下游 Kubernetes 集群，而无需通过 Rancher Server 进行身份验证。本文旨在为[设置 kubectl 以直接访问集群的说明](use-kubectl-and-kubeconfig.md#直接使用下游集群进行身份验证)提供背景信息和上下文。 -### kubeconfig 文件说明 +### Kubeconfig 文件说明 -_kubeconfig 文件_ 是与 kubectl 命令行工具（或其他客户端）结合使用时用于配置 Kubernetes 访问的文件。 +kubeconfig 文件是与 kubectl 命令行工具（或其他客户端）结合使用时用于配置 Kubernetes 访问的文件。 -此 kubeconfig 文件及其内容特定于你正在查看的集群。你可以从 Rancher 的**集群**视图中下载该文件。在 Rancher 中可以访问的每个集群都需要一个单独的 kubeconfig 文件。 +kubeconfig 文件及其内容特定于各个集群。你可以从 Rancher 的**集群**页面进行下载： + +1. 点击左上角的 **☰**。 +1. 选择**集群管理**。 +1. 找到要下载其 kubeconfig 的集群，然后选择行末尾的 **⁝**。 +1. 从子菜单中选择**下载 KubeConfig**。 + +在 Rancher 中可以访问的每个集群都需要一个单独的 kubeconfig 文件。 下载 kubeconfig 文件后，你将能够使用 kubeconfig 文件及其 Kubernetes [上下文](https://kubernetes.io/docs/reference/kubectl/cheatsheet/#kubectl-context-and-configuration)访问下游集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md index c28853ad3e4..ac6a4a338fb 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md @@ -27,9 +27,10 @@ description: "了解如何通过 kubectl Shell 使用 kubectl，或通过 kubect ::: -1. 登录到 Rancher。点击 **☰ > 集群管理**。 -1. 转到要使用 kubectl 访问的集群，然后单击 **Explore**。 -1. 在顶部导航栏中，单击**下载 KubeConfig** 按钮。 +1. 点击左上角的 **☰**。 +1. 选择**集群管理**。 +1. 找到要下载其 kubeconfig 的集群，然后选择行末尾的 **⁝**。 +1. 从子菜单中选择**下载 KubeConfig**。 1. 将 YAML 文件保存在本地计算机上。将文件移动到 `~/.kube/config`。注意：kubectl 用于 kubeconfig 文件的默认位置是 `~/.kube/config`。但是你也可以运行类似以下的命令，使用 `--kubeconfig` 标志指定任何其他目录： ``` kubectl --kubeconfig /custom/path/kube.config get pods diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md new file mode 100644 index 00000000000..1d76f44b72c --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -0,0 +1,79 @@ +--- +title: "Kubernetes 持久存储：卷和存储类" +description: "了解在 Kubernetes 中创建持久存储的两种方法：持久卷和存储类" +--- + + + + + +在部署需要保​​留数据的应用时，你需要创建持久存储。持久存储允许你在运行应用的 pod 之外存储应用数据。即使运行应用的 pod 发生故障，这种存储方式也能让你保留应用数据。 + +本文假设你已了解 Kubernetes 的持久卷、持久卷声明和存储类的概念。如需更多信息，请参阅[存储的工作原理](manage-persistent-storage/about-persistent-storage.md)部分。 + +### 先决条件 + +设置持久存储需要`管理卷`的[角色](../../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色参考)。 + +如果你要为云集群配置存储，则存储和集群主机必须使用相同的云提供商。 + +要使用 Rancher 配置新存储，则必须启用云提供商。有关启用云提供商的详细信息，请参阅[此页面](../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md)。 + +如果要将现有的持久存储连接到集群，则不需要启用云提供商。 + +### 设置现有存储 + +设置现有存储的总体流程如下： + +1. 设置你的持久存储。可以是云存储或你自己的存储。 +2. 添加引用持久存储的持久卷 (PV)。 +3. 添加引用 PV 的持久卷声明 (PVC)。 +4. 将 PVC 挂载为工作负载中的卷。 + +有关详细信息和先决条件，请参阅[此页面](manage-persistent-storage/set-up-existing-storage.md)。 + +### 在 Rancher 中动态配置新存储 + +配置新存储的总体流程如下： + +1. 添加一个 StorageClass 并将它配置为使用你的存储提供商。StorageClass 可以引用云存储或你自己的存储。 +2. 添加引用存储类的持久卷声明 (PVC)。 +3. 将 PVC 挂载为工作负载的卷。 + +有关详细信息和先决条件，请参阅[此页面](manage-persistent-storage/dynamically-provision-new-storage.md)。 + +### Longhorn 存储 + +[Longhorn](https://longhorn.io/) 是一个轻量级、可靠、易用的 Kubernetes 分布式块存储系统。 + +Longhorn 是免费的开源软件。Longhorn 最初由 Rancher Labs 开发，现在正在作为云原生计算基金会的沙盒项目进行开发。它可以通过 Helm、kubectl 或 Rancher UI 安装在任何 Kubernetes 集群上。 + +如果你有块存储池，Longhorn 可以帮助你为 Kubernetes 集群提供持久存储，而无需依赖云提供商。有关 Longhorn 功能的更多信息，请参阅[文档](https://longhorn.io/docs/latest/what-is-longhorn/)。 + +Rancher v2.5 简化了在 Rancher 管理的集群上安装 Longhorn 的过程。详情请参见[本页面](../../../../integrations-in-rancher/longhorn/longhorn.md)。 + +### 配置存储示例 + +我们提供了如何使用 [NFS](../provisioning-storage-examples/nfs-storage.md)， [vSphere](../provisioning-storage-examples/vsphere-storage.md)，和 [Amazon 的 EBS](../provisioning-storage-examples/persistent-storage-in-amazon-ebs.md) 来配置存储的示例。 + +### GlusterFS 卷 + +在将数据存储在 GlusterFS 卷上的集群中，你可能会遇到重启 `kubelet` 后 pod 无法挂载卷的问题。有关避免此情况发生的详细信息，请参阅[此页面](manage-persistent-storage/about-glusterfs-volumes.md)。 + +### iSCSI 卷 + +在将数据存储在 iSCSI 卷上的 [Rancher 启动的 Kubernetes 集群](../../launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)中，你可能会遇到 kubelet 无法自动连接 iSCSI 卷的问题。有关解决此问题的详细信息，请参阅[此页面](manage-persistent-storage/install-iscsi-volumes.md)。 + +### hostPath 卷 + +在创建 hostPath 卷之前，你需要在集群配置中设置 [extra_bind](https://rancher.com/docs/rke/latest/en/config-options/services/services-extras/#extra-binds/)。这会将路径作为卷安装在你的 kubelet 中，可用于工作负载中的 hostPath 卷。 + +### 将 vSphere Cloud Provider 从树内迁移到树外 + +Kubernetes 正在逐渐不在树内维护云提供商。vSphere 有一个树外云提供商，可通过安装 vSphere 云提供商和云存储插件来使用。 + +有关如何从树内 vSphere 云提供商迁移到树外，以及如何在迁移后管理现有虚拟机，请参阅[此页面](../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 + +### 相关链接 + +- [Kubernetes 文档: 存储](https://kubernetes.io/docs/concepts/storage/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/install-cluster-autoscaler/install-cluster-autoscaler.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/install-cluster-autoscaler/install-cluster-autoscaler.md new file mode 100644 index 00000000000..edbea4b5685 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/install-cluster-autoscaler/install-cluster-autoscaler.md @@ -0,0 +1,28 @@ +--- +title: Cluster Autoscaler +--- + + + + + +在本文中，你将学习如何使用 AWS EC2 Auto Scaling 组在 Rancher 自定义集群上安装和使用 [Kubernetes cluster-autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/)。 + +Cluster Autoscaler 是一个自动调整 Kubernetes 集群大小的工具。该工具在满足以下条件之一时能自动调整集群大小： + +* 集群中有 Pod 因资源不足而无法运行。 +* 集群中有一些节点长时间未得到充分利用，而且它们的 Pod 可以放到其他现有节点上。 + +为防止你的 pod 被驱逐，请在你的 pod 规范中设置 `priorityClassName: system-cluster-critical` 属性。 + +Cluster Autoscaler 运行在 Kubernetes master 节点上。它可以在 `kube-system` 命名空间中运行。Cluster Autoscaler 不会缩减运行非镜像 `kube-system` pod 的节点。 + +你可以在 worker 节点上运行 Cluster Autoscaler 的自定义 deployment，但需要小心以保证 Cluster Autoscaler 能正常运行。 + +## 云提供商 + +Cluster Autoscaler 为不同的云提供商提供支持。有关详细信息，请参见 [Cluster Autoscaler 支持的云提供商](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#deployment)。 + +### 在亚马逊云提供商上设置 Cluster Autoscaler + +有关在 Amazon 上运行 Cluster Autoscaler 的详细信息，请参阅[此页面](use-aws-ec2-auto-scaling-groups.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/manage-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/manage-clusters.md new file mode 100644 index 00000000000..99616d4ac2b --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/manage-clusters.md @@ -0,0 +1,36 @@ +--- +title: 集群管理 +--- + + + + + +在 Rancher 中配置集群后，你可以开始使用强大的 Kubernetes 功能在开发、测试或生产环境中部署和扩展容器化应用。 + +:::note + +本节默认你已对 Docker 和 Kubernetes 有一定的了解。如果你需要了解 Kubernetes 组件如何协作，请参见 [Kubernetes 概念](../../../reference-guides/kubernetes-concepts.md)页面。 + +::: + +## 在 Rancher 中管理集群 + +将集群[配置到 Rancher](../kubernetes-clusters-in-rancher-setup/kubernetes-clusters-in-rancher-setup.md) 之后，[集群所有者](../authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#cluster-roles)需要管理这些集群。管理集群的选项如下： + +import ClusterCapabilitiesTable from '../../../shared-files/_cluster-capabilities-table.md'; + + + +## Configuring Tools + +Rancher 包含 Kubernetes 中未包含的各种工具来协助你进行 DevOps 操作。Rancher 可以与外部服务集成，让你的集群更高效地运行。工具分为以下几类： + +- 告警 +- Notifiers +- Logging +- Monitoring +- Istio 服务网格 +- OPA Gatekeeper + +你可以通过 **Apps** 来安装工具。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/nodes-and-node-pools.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/nodes-and-node-pools.md index 2174397f7ce..a942534d1c6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/nodes-and-node-pools.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/nodes-and-node-pools.md @@ -2,7 +2,14 @@ title: 节点和节点池 --- -在 Rancher 中启动 Kubernetes 集群后，你可以从集群的**节点**选项卡管理各个节点。不同的配置集群[选项](../../../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)对应不同的可用节点选项。 +在 Rancher 中启动 Kubernetes 集群后，你可以从集群的**节点**选项卡管理各个节点。 + +1. 点击左上角的 **☰**。 +1. 选择**集群管理**。 +1. 找到要管理其节点的集群，然后单击行末尾的**浏览**按钮。 +1. 从左侧导航中选择**节点**。 + +不同的集群配置[选项](../../../pages-for-subheaders/kubernetes-clusters-in-rancher-setup.md)对应不同的可用节点选项。 :::note diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/provisioning-storage-examples.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/provisioning-storage-examples.md new file mode 100644 index 00000000000..2509db22979 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/provisioning-storage-examples.md @@ -0,0 +1,15 @@ +--- +title: 配置存储示例 +--- + + + + + +Rancher 通过各种卷插件支持持久化存储。不过，在使用这些插件将持久存储绑定到工作负载之前，您必须先配置存储本身，无论是服务提供商提供的基于云的解决方案，还是您自己管理的内部部署解决方案。 + +Rancher 提供了有关如何配置一些常用存储方法的文档： + +- [NFS](nfs-storage.md) +- [vSphere](vsphere-storage.md) +- [EBS](persistent-storage-in-amazon-ebs.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/new-user-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/new-user-guides.md new file mode 100644 index 00000000000..91cfa77fe1d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/how-to-guides/new-user-guides/new-user-guides.md @@ -0,0 +1,13 @@ +--- +title: 新用户指南 +--- + + + + + +新用户指南，也称为 **教程**，描述了用户在完成一些具体操作时要遵循的实用步骤。这些文档被称为“以学习为导向”的文档，用户通过“实际操作”来学习。 + +新用户指南旨在指导初学者或 Rancher 的日常用户通过一系列步骤来学习如何做某件事。目标是让用户可以通过易于操作、有意义的、可重复的指示来学习如何完成任务。这些指南将帮助用户完成工作，然后立即获得承诺的结果。 + +普通的 Rancher 用户的技术水平高于“初学者”，但是新用户指南旨在同等的帮助新用户、初学者以及经验丰富的 Rancher 客户，这是通过结合使用高级和技术语言来介绍主题并指导用户完成每个 Rancher 用户都必须了解的一般任务来实现的。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cis-scans/cis-scans.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cis-scans/cis-scans.md new file mode 100644 index 00000000000..41ca4d923cb --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cis-scans/cis-scans.md @@ -0,0 +1,121 @@ +--- +title: CIS 扫描 +--- + + + + + +Rancher 可以通过运行安全扫描来检查 Kubernetes 是否按照 CIS Kubernetes Benchmark 中定义的安全最佳实践进行部署。CIS 扫描可以运行在任何 Kubernetes 集群，包括托管的 Kubernetes，例如 EKS、AKS 和 GKE。 + +`rancher-cis-benchmark` 应用使用了 kube-bench, ，这是 Aqua Security 的开源工具，用于检查集群是否符合 CIS Kubernetes Benchmark。此外，为了生成集群级别的报告，此应用使用了 Sonobuoy 来聚合报告。 + +## 关于 CIS Benchmark + +CIS（Center for Internet Security）是一个 501(c\)(3) 非营利组织，成立于 2000 年 10 月，其使命是识别、开发、验证、促进和维持网络防御的最佳实践方案，并建立和指导社区，以在网络空间中营造信任的环境。该组织总部位于纽约东格林布什，其成员包括大公司、政府机构和学术机构。 + +CIS Benchmark 是目标系统安全配置的最佳实践。CIS Benchmark 是由安全专家、技术供应商、公开和私人社区成员，以及 CIS Benchmark 开发团队共同志愿开发的。 + +在 CIS 网站上[注册](https://learn.cisecurity.org/benchmarks)以查看官方 Benchmark 文档。 + +## 关于生成的报告 + +每次扫描都会生成一份报告，你可以在 Rancher UI 中查看该报告，并以 CSV 格式下载它。 + +默认情况下使用 CIS Benchmark v1.6。 + +Benchmark 版本包含在生成的报告中。 + +Benchmark 提供两种类型的建议，分别是自动（Automated）和手动（Manual）。Benchmark 中标记为 Manual 的建议不包含在生成的报告中。 + +一些测试会被标记为“不适用”。由于 Rancher 配置 RKE 集群的方式，这些测试不会在任何 CIS 扫描中运行。有关如何审核测试结果，以及为什么某些测试会被标记为不适用，请参阅 Rancher 的 Kubernetes 对应版本的[自测指南](../../reference-guides/rancher-security/rancher-security.md#CIS-Benchmark-和自我评估)。 + +该报告包含以下信息： + +| 报告中的列 | 描述 | +|-------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | CIS Benchmark 的 ID 号。 | +| `description` | CIS Benchmark 测试的描述。 | +| `remediation` | 为了通过测试需要修复的内容。 | +| `state` | 测试的状态，可以是通过、失败、跳过或不适用。 | +| `node_type` | 节点角色，角色决定了在节点上运行的测试。主测试在 controlplane 节点上运行，etcd 测试在 etcd 节点上运行，节点测试在 Worker 节点上运行。 | +| `audit` | 这是 `kube-bench` 为此测试运行的审计检查。 | +| `audit_config` | 适用于审计脚本的任何配置。 | +| `test_info` | `kube-bench` 报告的测试相关信息（如果存在）。 | +| `commands` | `kube-bench` 报告的测试相关的命令（如果存在）。 | +| `config_commands` | `kube-bench` 报告的测试相关的配置数据（如果存在）。 | +| `actual_value` | 测试的实际值。如果由 `kube-bench` 报告，则会显示。 | +| `expected_result` | 测试的预期值。如果由 `kube-bench` 报告，则会显示。 | + +请参阅[集群加固指南中的表格](../../reference-guides/rancher-security/rancher-security.md)，以了解 Kubernetes、Benchmark、Rancher 以及我们的集群强化指南的版本对应关系。另外，请参阅强化指南，以获取符合 CIS 的集群的配置文件以及修复失败测试的信息。 + +## 测试配置文件 + +以下是可用的配置文件： + +- Generic CIS 1.6 +- Generic CIS 1.20 +- Generic CIS 1.23 +- RKE permissive 1.6 +- RKE hardened 1.6 +- RKE permissive 1.20 +- RKE hardened 1.20 +- RKE permissive 1.23 +- RKE hardened 1.23 +- RKE2 permissive 1.6 +- RKE2 hardened 1.6 +- RKE2 permissive 1.20 +- RKE2 hardened 1.20 +- RKE2 permissive 1.23 +- RKE2 hardened 1.23 +- K3s permissive 1.6 +- K3s hardened 1.6 +- K3s permissive 1.20 +- K3s hardened 1.20 +- K3s permissive 1.23 +- K3s hardened 1.23 +- AKS +- EKS +- GKE + +你还可以通过保存一组要跳过的测试来自定义配置文件。 + +所有配置文件都会有一组不适用的测试，CIS 扫描会跳过这些测试。RKE 集群管理 Kubernetes 的方式导致这些测试被认为不适用。 + +RKE 集群扫描配置文件有两种类型： + +- **Permissive**：此配置文件有一组要跳过的测试，跳过的原因是这些测试会在默认的 RKE Kubernetes 集群上失败。除了跳过的测试列表之外，配置文件也不会运行不适用的测试。 +- **Hardened**：此配置文件不会跳过任何测试（不适用的测试除外）。 + +EKS 和 GKE 集群扫描的配置文件基于这些集群类型特定的 CIS Benchmark 版本。 + +要通过 “Hardened” 配置文件，你需要遵从[强化指南](../../reference-guides/rancher-security/rancher-security.md#Rancher-加固指南)并使用强化指南中定义的 `cluster.yml` 来配置一个强化集群。 + +默认配置文件和支持的 CIS Benchmark 版本取决于扫描的集群类型： + +`rancher-cis-benchmark` 支持 CIS 1.6 Benchmark 版本。 + +- RKE Kubernetes 集群默认使用 RKE Permissive 1.6 配置文件。 +- EKS 和 GKE 有自己的 CIS Benchmark，由 `kube-bench` 发布。这些集群默认使用相应的测试配置文件。 +- RKE2 Kubernetes 集群默认使用 RKE2 Permissive 1.6 配置文件。 +- RKE、RKE2、EKS 和 GKE 以外的集群类型默认使用 Generic CIS 1.5 配置文件。 + +## 跳过和不适用的测试 + +有关要跳过和不适用的测试列表，请参阅[此页面](../../how-to-guides/advanced-user-guides/cis-scan-guides/skip-tests.md)。 + +目前，只有用户定义的跳过测试会在生成报告中标记为跳过。 + +如果某个默认配置文件将某个测试定义为跳过，则该测试也会标记为不适用。 + +## RBAC + +有关权限的详细信息，请参阅[此页面](rbac-for-cis-scans.md)。 + +## 配置 + +有关为扫描、配置文件和 Benchmark 版本配置自定义资源的更多信息，请参阅[此页面](configuration-reference.md)。 + +## 操作指南 + +要了解如何运行 CIS 扫描，请参阅 [CIS 扫描指南](../../how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/aws-cloud-marketplace.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/aws-cloud-marketplace.md new file mode 100644 index 00000000000..c7dd9264ee4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/aws-cloud-marketplace.md @@ -0,0 +1,36 @@ +--- +title: AWS Marketplace 集成 +--- + + + + + +## 概述 + +Rancher 提供与 AWS Marketplace 的集成，允许用户购买 SUSE 的支持合同。当你开始支持更多集群时，这种集成使你可以轻松调整支持需求。 + +## 限制 + +- 你必须运行 Rancher v2.6.7 或更高版本。 +- Rancher 的部署必须启用额外的指标。 +- Rancher 必须安装在 EKS 集群上。 +- 你必须通过 AWS Marketplace 购买至少一项 Rancher 支持的权限。 +- 你可能需要额外的设置来支持代理/离线用例。有关更多信息，请参阅[先决条件](adapter-requirements.md)。 + +## 如何使用 + +1. 完成[先决条件](adapter-requirements.md)。 +2. [安装 CSP Adapter](install-adapter.md)。 + +## 常见问题 + +**我以后可以购买对更多节点的支持吗？** + +可以。只需转到你最初用于购买支持的 AWS Marketplace 条目并增加权限数量即可。 + +**我可以在同一个 AWS 账户中使用多个 Rancher 实例吗？** + +可以。但是，Rancher 安装的每个集群都需要遵守先决条件。 + +此外，一个给定的权限一次只能由一个 Rancher 管理服务器使用。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/install-adapter.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/install-adapter.md index 08bae7cbe2b..a3c86f81ec0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/install-adapter.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cloud-marketplace/aws-cloud-marketplace/install-adapter.md @@ -2,6 +2,10 @@ title: 安装 Adapter --- + + + + > **重要提示**：如果你尝试重新安装 Adapter，你可能会在长达一小时的时间内收到不合规的错误消息。 ### Rancher 与 Adapter 的兼容性矩阵 @@ -14,14 +18,20 @@ title: 安装 Adapter ::: | Rancher 版本 | Adapter 版本 | -|-----------------|:---------------:| -| v2.7.0 | v2.0.0 | -| v2.7.1 | v2.0.0 | -| v2.7.2 | v2.0.1 | -| v2.7.3 | v2.0.1 | -| v2.7.4 | v2.0.1 | -| v2.7.5 | v2.0.2 | - +| ------------ | :----------: | +| v2.7.0 | v2.0.0 | +| v2.7.1 | v2.0.0 | +| v2.7.2 | v2.0.1 | +| v2.7.3 | v2.0.1 | +| v2.7.4 | v2.0.1 | +| v2.7.5 | v2.0.2 | +| v2.7.6 | v2.0.2 | +| v2.7.7 | v2.0.2 | +| v2.7.8 | v2.0.2 | +| v2.7.9 | v2.0.2 | +| v2.7.10 | v2.0.2 | +| v2.7.11 | v2.0.4 | +| v2.7.12 | v2.0.4 | ### 1. 获取对 Local 集群的访问权限 @@ -88,7 +98,6 @@ helm repo add rancher-charts https://charts.rancher.io helm install rancher-csp-adapter rancher-charts/rancher-csp-adapter --namespace cattle-csp-adapter-system --set aws.enabled=true --set aws.roleName=$MY_ROLE_NAME --set-string aws.accountNumber=$MY_ACC_NUM --version $CSP_ADAPTER_VERSION ``` - 你也可以使用 `values.yaml` 并指定以下选项： ```yaml @@ -152,4 +161,4 @@ kubectl delete secret tls-ca-additional -n cattle-csp-adapter-system kubectl rollout restart deploy rancher-csp-adapter -n cattle-csp-adapter-system ``` -> **注意**：有一些方法（例如 cert-manager 的 [trust operator](https://cert-manager.io/docs/projects/trust/)）可以帮助你减少手动轮换任务的数量。这些选项不受官方支持，但可能对想要自动化某些任务的用户有用。 \ No newline at end of file +> **注意**：有一些方法（例如 cert-manager 的 [trust operator](https://cert-manager.io/docs/projects/trust/)）可以帮助你自动执行一些任务。尽管这些方法不受官方支持，但它们可以减少手动轮换任务的频率。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cloud-marketplace/cloud-marketplace.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cloud-marketplace/cloud-marketplace.md new file mode 100644 index 00000000000..44c95be8fe9 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cloud-marketplace/cloud-marketplace.md @@ -0,0 +1,11 @@ +--- +title: 云市场集成 +--- + + + + + +Rancher 提供与云市场的集成，以便轻松购买对托管在某些云提供商上的安装支持。此外，此集成还提供了生成 Supportconfig Bundle 的功能，该 Bundle 可以给 rancher 提供支持。 + +此集成仅支持 AWS。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cluster-api/cluster-api.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cluster-api/cluster-api.md new file mode 100644 index 00000000000..1bd4160152e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cluster-api/cluster-api.md @@ -0,0 +1,14 @@ +--- +title: Cluster API (CAPI) 与 Rancher Turtles +--- + + + + + +[Rancher Turtles](https://turtles.docs.rancher.com/) 是一个 [Rancher 扩展](../rancher-extensions.md)，通过提供 Cluster API (CAPI) 和 Rancher 之间的集成来管理配置的 Kubernetes 集群的生命周期。使用 Rancher Turtles，你可以： + +- 通过在 CAPI 配置的集群中安装 Rancher Cluster Agent，将 CAPI 集群导入 Rancher。 +- 配置 [CAPI Operator](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values)。 + +[概述](./overview.md)部分介绍了安装选项、Rancher Turtles 架构和简要 Demo。有关详细信息，请参阅 [Rancher Turtles 文档](https://turtles.docs.rancher.com/)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cluster-api/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cluster-api/overview.md new file mode 100644 index 00000000000..5da4470bfd6 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/cluster-api/overview.md @@ -0,0 +1,260 @@ +--- +title: 概述 +--- + + + + + +## 架构图 + +下面是 Rancher Turtles 的关键组件及其与 Rancher 和 Rancher Cluster Agent 的关系的架构图，了解这些组件对于深入了解 Rancher 如何利用 CAPI operator 进行集群管理至关重要。 + +![overview](/img/30000ft_view.png) + +## 安全 + +[SLSA](https://slsa.dev/spec/v1.0/about) 是一套由行业共识制定的可逐步采用的供应链安全指南。SLSA 制定的规范对软件生产者和消费者都很有用：生产者可以遵循 SLSA 的指导方针，使他们的软件供应链更加安全，消费者可以使用 SLSA 来决定是否信任软件包。 + +Rancher Turtles 满足 [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) 对适当的构建平台、一致的构建过程和来源分布的要求。更多信息请参阅 [Rancher Turtles 安全](https://turtles.docs.rancher.com/security/slsa)文档。 + +## 先决条件 + +在 Rancher 环境中安装 Rancher Turtles 之前，你必须禁用 Rancher 的 `embedded-cluster-api` 功能。这还包括清理 Rancher 专用的 webhook，否则这些 webhook 将与 CAPI 的 webhook 冲突。 + +为了简化 Rancher 安装 Rancher Turtles 的设置，官方的 Rancher Turtles Helm chart 包含一个删除以下内容的 `pre-install` hook： + +- 禁用 Rancher 中的 `embedded-cluster-api` 功能。 +- 删除不再需要的 `mutating-webhook-configuration` 和 `validating-webhook-configuration` webhook。 + +这些 webhook 也可以通过 Rancher UI 删除： + +1. 点击左上角 **☰ > 集群管理**。 +1. 选择你的 local 集群。 +1. 在左侧导航菜单，选择 **More Resources** > **Admission**。 +1. 在下拉菜单中，选择资源页面的 `MutatingWebhookConfiguration` 和 `ValidatingWebhookConfiguration`。 +1. 在相应的资源页面上，点击 `mutating-webhook-configuration` and `validating-webhook-configuration` 后面的 **⋮** 然后选择 **删除**。 + +还可以通过在 **Resource Search** 字段中输入 webhook 的名称来访问到具体的 webhook。 + +以下的 `kubectl` 命令可以手动删除必要的 webhook： + +```console +kubectl delete mutatingwebhookconfiguration.admissionregistration.k8s.io mutating-webhook-configuration +``` + +```console +kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io validating-webhook-configuration +``` + +使用以下示例从控制台禁用 `embedded-cluster-api` 功能： + +1. 创建一个 `feature.yaml` 文件，将 `embedded-cluster-api` 设置为 false： + +```yaml title="feature.yaml" +apiVersion: management.cattle.io/v3 +kind: Feature +metadata: + name: embedded-cluster-api +spec: + value: false +``` + +2. 使用 `kubectl` 将 `feature.yaml` 文件应用到集群： + +```bash +kubectl apply -f feature.yaml +``` + +## 安装 Rancher Turtles Operator + +你可以通过 Rancher UI 或使用 Helm 安装 Rancher Turtles operator。对于大多数环境推荐使用第一种方法。 + +:::caution + +如果你的集群中已经安装了 Cluster API (CAPI) Operator，你必须使用[手动 Helm 安装方法](#通过-helm-安装)。 + +::: + +### 通过 Rancher UI 安装 + +通过 Rancher UI 添加 Turtles 仓库，Rancher 可以处理 CAPI 扩展的安装和配置。 + +1. 点击 **☰**。在左侧导航栏**浏览集群**的菜单下，选择 **local**。 +1. 在 **Cluster Dashboard** 的左侧导航菜单中，点击 **Apps > Repositories**。 +1. 点击 **Create** 创建一个新的仓库。 +1. 输入以下信息： + - **Name**: turtles + - **Index URL**: https://rancher.github.io/turtles +1. 等待新的仓库状态更新为 `Active`。 +1. 在左侧导航菜单中，点击 **Apps > Charts**。 +1. 在搜索过滤器中输入 "turtles" 来查找 Turtles chart。 +1. 点击 **Rancher Turtles - the Cluster API Extension**。 +1. 点击 **Install > Next > Install**. + +此过程使用 Helm chart 的默认值，这些值适用于大部分的安装场景。如果你的配置需要覆盖其中一些默认值，你可以在安装期间通过 Rancher UI 指定这些值，也可以通过 [Helm 手动安装 Chart](#通过-helm-安装)。有关可用的 values 设置的详细信息，请参阅 Rancher Turtles 的 [Helm chart 参考指南](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values)。 + +安装可能需要几分钟时间，安装完成后，你可以在集群中看到以下新部署： + +- `rancher-turtles-system/rancher-turtles-controller-manager` +- `rancher-turtles-system/rancher-turtles-cluster-api-operator` +- `capi-system/capi-controller-manager` + +#### Demo + +这个 demo 演示了如何使用 Rancher UI 安装 Rancher Turtles、创建/导入一个 CAPI 集群，以及在集群上安装监控： + + + +### 通过 Helm 安装 + +通过 Helm 安装 Rancher Turtles 有两种方法，这取决于你是否将 CAPI operator 作为依赖项包含其中： + +- [使用 CAPI Operator 作为依赖项安装 Rancher Turtles](#使用-cluster-api-capi-operator-作为-helm-依赖项安装-rancher-turtles)。 +- [安装没有 CAPI Operator 的 Rancher Turtles](#不使用-cluster-api-capi-operator-作为-helm-依赖安装-rancher-turtles)。 + +安装 Rancher Turtles 需要 CAPI Operator。你可以选择自己处理此依赖项，还是让 Rancher Turtles Helm chart 替你管理它。[使用 CAPI Operator 作为依赖项安装 Rancher Turtles](#使用-cluster-api-capi-operator-作为-helm-依赖项安装-rancher-turtles) 更简单，但是你的最佳选择取决于你的具体配置。 + +CAPI Operator 允许使用声明式方法处理 CAPI provider 的生命周期，扩展了 `clusterctl` 的能力。如果你想了解更多相关内容，可以参考 [Cluster API Operator book](https://cluster-api-operator.sigs.k8s.io/)。 + +#### 使用 `Cluster API (CAPI) Operator` 作为 Helm 依赖项安装 Rancher Turtles + +1. 添加包含 `rancher-turtles` chart 的 Helm 仓库作为安装的第一步： + +```bash +helm repo add turtles https://rancher.github.io/turtles +helm repo update +``` + +2. 如前面所述，安装 Rancher Turtles 需要 [CAPI Operator](https://github.com/kubernetes-sigs/cluster-api-operator)。Helm chart 可以使用一组最少的参数自动安装： + +```bash +helm install rancher-turtles turtles/rancher-turtles --version \ + -n rancher-turtles-system \ + --dependency-update \ + --create-namespace --wait \ + --timeout 180s +``` + +3. 此操作可能需要几分钟时间，完成后，你可以查看下面列出的已安装的控制器： + +- `rancher-turtles-controller` +- `capi-operator` + +:::note + +- 如果集群中已经有可用的 `cert-manager`，请在安装时通过以下参数将 Rancher Turtles 的此项依赖禁用掉，来阻止冲突： + `--set cluster-api-operator.cert-manager.enabled=false` +- 有关 Rancher Turtles 的版本列表，请参阅 [Turtles 发布页面](https://github.com/rancher/turtles/releases)。 + +::: + +这是最基本的推荐配置，用于管理在核心提供商的命名空间中创建包含所需 CAPI 功能标志（已启用 `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` 和 `EXP_MACHINE_POOL` ）的 secret。要启用其他 CAPI 功能需要启用上述功能标志。 + +如果你需要覆盖默认的行为并使用现有 secret 或添加自定义环境变量，你可以将 secret 名称通过 Helm 参数传入。在这种情况下，你作为负责管理 secret 创建和内容的用户，需要启用的最少特性包括：`CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` 和 `EXP_MACHINE_POOL`。 + +```bash +helm install ... + # Passing secret name and namespace for additional environment variables + --set cluster-api-operator.cluster-api.configSecret.name= +``` + +以下是一个用户管理 secret `cluster-api-operator.cluster-api.configSecret.name=variables` 的示例，其中设置了 `CLUSTER_TOPOLOGY`, `EXP_CLUSTER_RESOURCE_SET` 和 `EXP_MACHINE_POOL` 功能以及一个额外的自定义变量： + +```yaml title="secret.yaml" +apiVersion: v1 +kind: Secret +metadata: + name: variables + namespace: rancher-turtles-system +type: Opaque +stringData: + CLUSTER_TOPOLOGY: "true" + EXP_CLUSTER_RESOURCE_SET: "true" + EXP_MACHINE_POOL: "true" + CUSTOM_ENV_VAR: "false" +``` + +:::info + +有关 chart 支持的 values 及其用法的详细信息，请参阅 [Helm chart 选项](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values) + +::: + +#### 不使用 `Cluster API (CAPI) Operator` 作为 Helm 依赖安装 Rancher Turtles + +:::note + +请记住，如果使用此安装选项，你必须自行管理 CAPI Operator 的安装。你可以参照 Rancher Turtles 文档中的 [CAPI Operator 指南](https://turtles.docs.rancher.com/tasks/capi-operator/intro) + +::: + +1. 添加包含 `rancher-turtles` chart 的 Helm 仓库作为安装的第一步： + +```bash +helm repo add turtles https://rancher.github.io/turtles +helm repo update +``` + +2. 将 chart 安装到 `rancher-turtles-system` 命名空间： + +```bash +helm install rancher-turtles turtles/rancher-turtles --version + -n rancher-turtles-system + --set cluster-api-operator.enabled=false + --set cluster-api-operator.cluster-api.enabled=false + --create-namespace --wait + --dependency-update +``` + +前面的命令告诉 Helm 忽略将 `cluster-api-operator` 作为依赖项安装。 + +3. 此操作可能需要几分钟，完成后你可以查看下面列出的已安装的控制器： + +- `rancher-turtles-controller` + +## 卸载 Rancher Turtles + +:::caution + +在 Rancher 环境中安装 Rancher Turtles 时，Rancher Turtles 默认会启用 CAPI Operator 清理。这包括清理 CAPI Operator 特定的 webhook 和 deployments，否则会导致 Rancher 配置出现问题。 + +为了简化 Rancher Turtles 卸载（通过 Rancher 或 Helm 命令），官方的 Rancher Turtles Helm chart 包含了一个删除以下内容的 `post-delete` hook： + +- 删除不再需要的 `mutating-webhook-configuration` 和 `validating-webhook-configuration` webhook。 +- 删除不再需要的 CAPI `deployments`。 + +::: + +卸载 Rancher Turtles： + +```bash +helm uninstall -n rancher-turtles-system rancher-turtles --cascade foreground --wait +``` + +这可能需要几分钟才能完成。 + +:::note + +请记住，如果你在安装时使用了不同的名称或不同的命名空间，你需要针对你的特定配置自定义卸载命令。 + +::: + +卸载 Rancher Turtles 后， Rancher 的 `embedded-cluster-api` 功能必须重新启用。 + +1. 创建一个 `feature.yaml` 文件，将 `embedded-cluster-api` 设置为 true： + +```yaml title="feature.yaml" +apiVersion: management.cattle.io/v3 +kind: Feature +metadata: + name: embedded-cluster-api +spec: + value: true +``` + +2. 使用 `kubectl` 将 `feature.yaml` 文件应用到集群： + +```bash +kubectl apply -f feature.yaml +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/elemental/elemental.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/elemental/elemental.md new file mode 100644 index 00000000000..6352de02ba1 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/elemental/elemental.md @@ -0,0 +1,30 @@ +--- +title: 使用 Elemental 进行操作系统管理 +--- + + + + + +Elemental 支持云原生主机管理。Elemental 允许你在任何位置装载任何机器，无论是在数据中心还是在边缘，并将它们无缝集成到 Kubernetes 中，同时管理你的工作流程（例如操作系统更新）。 + +## Elemental 和 Rancher + +Rancher 中的 Elemental： + +- 是 Kubernetes 原生的，它允许你通过 Kubernetes 集群中的 Elemental 管理操作系统。 +- 从 Kubernetes 的操作角度来看，不会造成干扰。 +- 是声明性的，并且 GitOps 友好。 +- 允许可信的、确定性的和可预测的 OCI Image-based flows。 +- 大规模运作。它支持 Fleet 规模的操作系统管理。 + +### 我什么时候应该使用 Elemental？ + +- Elemental 通过 Rancher manager 实现云原生操作系统管理。它适用于任何操作系统（例如，SLE Micro vanilla）。 +- Elemental 允许对数据中心和边缘的机器进行云原生管理。 +- Elemental 非常灵活，允许平台团队在其机器群中执行各种工作流。 + +## Elemental 和 Rancher Prime + +- 已作为 GUI 扩展深度集成到 Rancher 中。 +- 将 Rancher 的用例扩展到操作系统，如今 SLE Micro 与 Rancher 可以完美配合。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/architecture.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/architecture.md new file mode 100644 index 00000000000..ffdbff3edc2 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/architecture.md @@ -0,0 +1,11 @@ +--- +title: 架构 +--- + + + + + +Fleet 可以管理来自 Git 的原始 Kubernetes YAML、Helm Chart、Kustomize 或三者的任何组合的部署。无论来源如何，所有资源都会动态转化为 Helm Chart，并使用 Helm 作为引擎来部署到集群中的所有内容。这为你提供了高度的控制力、一致性和可审计性。Fleet 不仅关注扩展能力，而且还提供高度的控制和可见性，从而让用户准确了解集群上安装的内容。 + +![架构](/img/fleet-architecture.svg) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/fleet.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/fleet.md new file mode 100644 index 00000000000..8b1249e41a0 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/fleet.md @@ -0,0 +1,27 @@ +--- +title: 使用 Fleet 进行持续交付 +--- + + + + + +Fleet 通过集群 Fleet 的供应链协调和管理应用程序的持续交付。Fleet 使用 GitOps 作为一种安全的运营模式，组织供应链，帮助团队及时自信地交付。 + +## Fleet 和 Rancher + +许多用户经常同时管理 10 个以上的集群。鉴于集群的激增，持续交付是 Rancher 的重要组成部分。Fleet 使用 GitOps 确保可靠的持续交付体验，这是一种安全且越来越常见的运营模式。 + +### 我什么时候应该使用 Fleet? + +- 我需要跨地理区域部署我的监控堆栈（例如 Grafana、Prometheus），每个区域都有不同的保留策略。 +- 我是一名平台运营商，希望使用可扩展且安全的操作模型（GitOps）为集群配置所有组件。 +- 我是一名应用程序开发人员，希望我的最新更改自动进入我的开发环境。 + +## Fleet 和 Rancher Prime + +Fleet 已经作为持续交付工具和 GitOps 引擎深度集成到 Rancher 中。 + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/overview.md new file mode 100644 index 00000000000..55b89b8ee08 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/overview.md @@ -0,0 +1,66 @@ +--- +title: 概述 +--- + + + + + +使用 Fleet 进行持续交付是大规模的 GitOps。Fleet 旨在管理多达一百万个集群。它也足够轻量级，对于[单个集群](https://fleet.rancher.io/installation#default-install)也很有效，但当你达到[大规模](https://fleet.rancher.io/installation#configuration-for-multi-cluster)时，它真的会大放异彩。大规模是指单个组织中的大量集群、大量部署或大量团队。 + +Fleet 是 Rancher 的一个独立项目，可以通过 Helm 安装在任何 Kubernetes 集群上。 + +## 架构 + +有关 Fleet 如何运作的信息，请参阅[架构](./architecture)页面。 + +## 在 Rancher UI 中访问 Fleet + +Fleet 预安装在 Rancher 中，并由 Rancher UI 中的**持续交付**选项进行管理。有关持续交付和其他 Fleet 故障排除技巧的更多信息，请参阅[此处](https://fleet.rancher.io/troubleshooting)。 + +用户可以按照 **gitops** 实践，利用持续交付将应用程序部署到 git 仓库中的 Kubernetes 集群，而无需任何手动操作。 + +按照以下步骤在 Rancher UI 中访问持续交付： + +1. 单击 **☰ > 持续交付**。 + +1. 在菜单顶部选择命名空间 ，注意以下事项： + + - 默认情况下，选择 **fleet-default**，包括通过 Rancher 注册的所有下游集群。 + + - 你可以切换到 **fleet-local**，它仅包含 **local** 集群，或者你可以创建自己的工作空间，将集群分配和移动到其中。 + - 然后，你可以通过单击左侧导航栏上的**集群**来管理集群。 + +1. 单击左侧导航栏上的 **Gitrepos**，将 gitrepo 部署到当前工作空间的集群中。 + +1. 选择 [git 仓库](https://fleet.rancher.io/gitrepo-add)和[目标集群/集群组](https://fleet.rancher.io/gitrepo-targets)。你也可以通过单击左侧导航栏中的**集群组**在 UI 中创建集群组。 + +1. 部署 gitrepo 后，你可以通过 Rancher UI 监控应用程序。 + +## Windows 支持 + +有关对具有 Windows 节点的集群的支持的详细信息，请参阅 [Windows 支持](./windows-support)页面。 + +## GitHub 仓库 + +Fleet Helm charts 可在[此处](https://github.com/rancher/fleet/releases)获取。 + +## 在代理后使用 Fleet + +有关在代理后面使用 Fleet 的详细信息，请参阅[在代理后使用 Fleet](./use-fleet-behind-a-proxy)页面。 + +## Helm Chart 依赖 + +为了成功部署具有依赖项的 Helm Chart，你必须运行一个手动命令（如下所示），因为用户需要满足依赖列表。如果不执行此操作，继续克隆你的仓库并运行 `helm install`，则会因依赖项丢失而导致安装失败。 + +git 仓库中的 Helm Chart 必须在 Chart 子目录中包含其依赖。 你必须手动运行 `helm dependencies update $chart` 或在本地运行 `helm dependencies build $chart`，然后将完整的 Chart 目录提交到 git 仓库。请注意，你需要使用适当的参数更新你的命令。 + +## 故障排除 + +- **已知问题**：Fleet gitrepos 的 clientSecretName 和 helmSecretName 密文不包含在 [backup-restore-operator](../../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher.md#1-install-the-rancher-backup-operator) 创建的备份或恢复中。一旦有永久的解决方案，我们将更新社区内容。 + +- **临时解决方法**：默认情况下，用户定义的密文不会在 Fleet 中备份。如果执行灾难恢复或将 Rancher 迁移到新集群，则有必要重新创建密文。要修改 ResourceSet 以包含要备份的额外资源，请参阅文档[此处](https://github.com/rancher/backup-restore-operator#user-flow)。 + +## 文档 + +Fleet 文档位于 https://fleet.rancher.io/ 。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md new file mode 100644 index 00000000000..3806cbb2248 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/use-fleet-behind-a-proxy.md @@ -0,0 +1,58 @@ +--- +title: 在代理后使用 Fleet +--- + + + + + +在本节中，你将学习如何在具有公共 IP 的 Rancher 服务器，以及一个没有公共 IP 但配置为使用代理的 Kubernetes 集群的设置中启用 Fleet。 + +Rancher 不会与已注册的下游集群建立连接。部署在下游集群上的 Rancher agent 必须能够与 Rancher 建立连接。 + +要让 Fleet 在代理后工作，你需要为下游集群设置 **Agent 环境变量**。以下是集群级别的配置选项。 + +你可以通过 Rancher UI 为任何集群类型（包括注册集群和自定义集群）配置这些环境变量。可以在编辑现有集群或配置新集群时添加变量。 + +对于公共下游集群，[在 Rancher UI 中设置必要的环境变量](#在-rancher-ui-中设置环境变量)就足够了。 + +对于私有节点或私有集群，则需要在节点上设置环境变量。然后，在配置自定义集群或注册私有集群时，在 Rancher UI 中配置环境变量。有关如何在 K3s 集群中的 Ubuntu 节点上设置环境变量的示例，请参阅[本节](#在私有节点上设置环境变量)。 + +## 必要的环境变量 + +为代理添加 Fleet agent 环境变量时，将 替换为你的私有代理 IP。 + +| 变量名称 | 值 | +| ------------- | ----------------------------------------------------------------------- | +| `HTTP_PROXY` | http://:8888 | +| `HTTPS_PROXY` | http://:8888 | +| `NO_PROXY` | 127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local | + +## 在 Rancher UI 中设置环境变量 + +要将环境变量添加到现有集群： + +1. 单击 **☰ > 集群管理**。 +1. 转到要添加环境变量的集群，然后单击 **⋮ > 编辑配置**。 +1. 单击**高级选项**。 +1. 单击**添加环境变量**。 +1. 输入[必要的环境变量](#必要的环境变量) +1. 单击**保存**。 + +**结果：** Fleet agent 会在代理后工作。 + +## 在私有节点上设置环境变量 + +对于私有节点和私有集群，需要在节点本身上设置代理环境变量，并在 Rancher UI 中进行配置。 + +此示例显示了如何在 K3s 集群中的 Ubuntu 节点上设置环境变量： + +``` +ssh -o ForwardAgent=yes ubuntu@ +ssh +export proxy_private_ip= +export HTTP_PROXY=http://${proxy_private_ip}:8888 +export HTTPS_PROXY=http://${proxy_private_ip}:8888 +export NO_PROXY=127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,.svc,.cluster.local +export KUBECONFIG=/etc/rancher/k3s/k3s.yaml +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/windows-support.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/windows-support.md new file mode 100644 index 00000000000..7102f0fa01d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/fleet/windows-support.md @@ -0,0 +1,25 @@ +--- +title: Windows 支持 +--- + + + + + +在 Rancher v2.5.6 之前，`agent` 在具有 Windows 节点的下游集群上没有原生的 Windows 清单。这将导致集群的 `agent` pod 运行失败。 + +如果你从旧版本的 Rancher 升级到 v2.5.6+，你可以在 _下游集群_ 中部署具有以下工作流的可运行的 `agent`： + +1. 封锁所有 Windows 节点。 +1. 对 `agent` 工作负载应用以下容忍度。 +1. 取消所有 Windows 节点的封锁。 +1. 删除所有 `agent` pod。使用新的容忍度来创建新 pod。 +1. 一旦 `agent` pod 运行，并且 Fleet 的自动更新已启用，它们会更新到兼容 Windows 的 `agent` 版本。 + +```yaml +tolerations: + - effect: NoSchedule + key: cattle.io/os + operator: Equal + value: linux +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/harvester/harvester.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/harvester/harvester.md new file mode 100644 index 00000000000..8f4b2312b56 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/harvester/harvester.md @@ -0,0 +1,15 @@ +--- +title: 使用 Harvester 在 Kubernetes 上进行虚拟化 +--- + + + + + +## Harvester + +Harvester 是 Rancher v2.6.1 新增的功能，是基于 Kubernetes 构建的开源超融合基础架构（HCI）软件。Harvester 安装在裸金属服务器上，提供集成的虚拟化和分布式存储功能。虽然 Harvester 使用 Kubernetes 运行，但它不需要用户了解 Kubernetes 概念，这使得它更加用户友好。 + +## Harvester 与 Rancher + +凭借 Rancher Prime 和 Harvester，IT 运维人员现在可以访问一个企业级的、易于使用的基础设施平台，该平台可以协同管理他们的虚拟机和 Kubernetes 集群。有关产品支持的更多信息，请参阅[支持矩阵](https://www.suse.com/suse-harvester/support-matrix/all-supported-versions/harvester-v1-2-0/)。通过 Rancher 虚拟化管理功能，用户可以导入和管理多个 Harvester 集群。利用 Rancher 的认证功能和 RBAC 控制来支持多租户。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/harvester/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/harvester/overview.md new file mode 100644 index 00000000000..ac720bec060 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/harvester/overview.md @@ -0,0 +1,42 @@ +--- +title: 概述 +--- + + + + + +[Harvester](https://docs.harvesterhci.io/) 是 Rancher v2.6.1 新增的功能，是基于 Kubernetes 构建的开源超融合基础架构（HCI）软件。Harvester 安装在裸金属服务器上，提供集成的虚拟化和分布式存储功能。虽然 Harvester 使用 Kubernetes 运行，但它不需要用户了解 Kubernetes 概念，这使得它更加用户友好。 + +### 功能开关 + +Harvester 功能开关用于管理对 Rancher 中虚拟化管理（VM）页面的访问，用户可以直接导航到 Harvester 集群并访问 Harvester UI。Harvester 的功能开关默认启用。如需了解 Rancher 中功能开关的更多详细信息，请单击[此处](../../how-to-guides/advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 + +要导航到 Harvester 集群，请单击 **☰ > 虚拟化管理**。在 Harvester 集群页面中，单击集群以转到该 Harvester 集群的视图。 + +- 如果启用了 Harvester 功能开关，Harvester 集群将会从列出 Kubernetes 集群的任何页面或应用（例如 Fleet 和多集群应用）中过滤掉。 + +- 如果禁用了 Harvester 功能开关，并且导入了 Harvester 集群，Harvester 集群将显示在集群管理页面的 Rancher 集群列表中。仅当功能开关为关闭时，Harvester 集群才会显示在集群列表中。 + +- 集成 Harvester 后，你可以将 Harvester 集群导入 Rancher，对应的集群类型是 `Harvester`。 + +- 用户只能在虚拟化管理页面上导入 Harvester 集群。不支持在集群管理页面上导入集群，并且会出现警告，建议你返回虚拟化管理页面执行此操作。 + +### Harvester 主机驱动 + +[Harvester 主机驱动](https://docs.harvesterhci.io/v1.1/rancher/node/node-driver/)通常可用于 Rancher 中的 RKE 和 RKE2 选项。无论 Harvester 功能开关是否启用，主机驱动都是可用的。请注意，主机驱动默认处于关闭状态。用户只能通过集群管理页面在 Harvester 上创建 RKE 或 RKE2 集群。 + +Harvester 允许通过 Harvester UI 上传和显示 `.ISO` 镜像，但 Rancher UI 是不支持的。这是因为 `.ISO` 镜像通常需要额外的设置，这会干扰干净的部署（即无需用户干预），并且它们通常不用于云环境。 + +如需了解 Rancher 中主机驱动的更多详细信息，请单击[此处](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/about-provisioning-drivers#主机驱动)。 + +### 端口要求 + +Harvester 集群的端口要求可以在[此处](https://docs.harvesterhci.io/v1.1/install/requirements#networking)找到。 + +此外，其他网络注意事项如下： + +- 请务必为 VM VLAN 网络启用物理交换机的 VLAN 中继端口。 +- 按照[此处](https://docs.harvesterhci.io/v1.1/networking/index)的网络设置指南进行操作。 + +对于其他集群（例如 K3s 和 RKE1）的其他端口要求，请参阅[这些文档](https://docs.harvesterhci.io/v1.1/install/requirements/#guest-clusters)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/integrations-in-rancher.mdx b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/integrations-in-rancher.mdx new file mode 100644 index 00000000000..c0a824cc9da --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/integrations-in-rancher.mdx @@ -0,0 +1,51 @@ +--- +title: Rancher 中的集成 +--- + + + + + +import { Card, CardSection } from "@site/src/components/CardComponents"; +import { RocketRegular } from "@fluentui/react-icons"; + +Prime 是 Rancher 生态系统的企业级产品，具有更高的安全性、更长的生命周期和对 Prime 专有文档的访问权限。Rancher Prime 安装资产托管在受信任的 SUSE 注册表上，由 Rancher 拥有和管理。受信任的 Prime 注册表仅包括经过社区测试的稳定版本。 + +Prime 还提供生产支持选项，以及根据你的商业需求定制的订阅附加组件。 + +要了解更多信息并开始使用 Rancher Prime，请访问[本页](https://www.rancher.com/quick-start)。 + +}> + + + + + + + + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/istio/configuration-options/configuration-options.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/istio/configuration-options/configuration-options.md new file mode 100644 index 00000000000..800b4e576da --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/istio/configuration-options/configuration-options.md @@ -0,0 +1,43 @@ +--- +title: 配置选项 +--- + + + + + +### Egress 支持 + +默认情况下，Egress 网关是禁用的，但你可以在安装或升级时使用 values.yaml 或[覆盖文件](#覆盖文件)启用它。 + +### 启用自动 Sidecar 注入 + +默认情况下，自动 sidecar 注入是禁用的。要启用此功能，请在安装或升级时在 values.yaml 中设置 `sidecarInjectorWebhook.enableNamespacesByDefault=true`。这会自动将 Istio sidecar 注入到所有已部署的新命名空间。 + +### 覆盖文件 + +覆盖文件用于为 Istio 进行更广泛的配置。它允许你更改 [IstioOperator API](https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/) 中可用的任何值。你可以自定义默认安装以满足你的需求。 + +覆盖文件将在 Istio Chart 默认安装的基础上添加配置。换言之，你不需要为安装中已定义的组件进行重新定义。 + +有关覆盖文件的更多信息，请参阅 [Istio 文档](https://istio.io/latest/docs/setup/install/istioctl/#configure-component-settings) + +### 选择器和抓取配置 + +Monitoring 应用设置了 `prometheus.prometheusSpec.ignoreNamespaceSelectors=false`，即在默认情况下跨所有命名空间进行监控。这样，你可以查看部署在具有 `istio-injection=enabled` 标签的命名空间中的资源的流量、指标和图。 + +如果你想将 Prometheus 限制为特定的命名空间，请设置 `prometheus.prometheusSpec.ignoreNamespaceSelectors=true`。完成此操作后，你需要添加其他配置来继续监控你的资源。 + +详情请参阅[本节](selectors-and-scrape-configurations.md)。 + +### 在具有 Pod 安全策略的情况下启用 Istio + +详情请参阅[本节](pod-security-policies.md)。 + +### 在 RKE2 集群上安装 Istio 的其他步骤 + +详情请参阅[本节](install-istio-on-rke2-cluster.md)。 + +### 项目网络隔离的其他步骤 + +详情请参阅[本节](project-network-isolation.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/istio/istio.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/istio/istio.md new file mode 100644 index 00000000000..42c642b4b64 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/istio/istio.md @@ -0,0 +1,140 @@ +--- +title: Istio +--- + + + + + +[Istio](https://istio.io/) 是一种开源工具，可以让 DevOps 团队更轻松地观察、控制、排查并保护复杂的微服务网络中的流量。 + +随着微服务网络的变化和增长，微服务网络之间的交互变得越来越难以管理和理解。在这种情况下，将服务网格作为单独的基础设施层是非常有用的。Istio 的服务网格可以让你在不直接更改微服务的情况下控制微服务之间的流量。 + +Rancher 与 Istio 集成，使得管理员或集群所有者可以将 Istio 交给开发者团队，然后开发者使用 Istio 执行安全策略，排查问题，或为蓝绿部署，金丝雀部署，和 A/B 测试进行流量管理。 + +此核心服务网格支持但不限于以下功能： + +- **管理流量**：例如入口和出口路由、断路、镜像。 +- **安全**：具有用于验证和授权流量和用户的资源，包括 mTLS。 +- **可观察性**：观察日志、指标和分布式流量。 + +[设置 Istio](../../how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md) 后，你可以通过 Rancher UI、`kubectl` 或 ` Istioctl` 来使用 Istio 的 controlplane 功能。 + +Istio 需要由 `cluster-admin` 设置后才能在项目中使用。 + +## Rancher 2.5 的新功能 + +Istio 已简化了整体架构。结合 Pilot、Citadel、Galley 和 sidecar injector 创建了一个单独的组件 Istiod。Node Agent 功能也已合并到 istio-agent 中。 + +以前由 Istio 安装的插件（cert-manager、Grafana、Jaeger、Kiali、Prometheus、Zipkin）现在需要单独安装。Istio 支持安装来自 Istio 项目的集成，并保持与非 Istio 项目的兼容性。 + +你仍然可以通过安装 [Rancher Monitoring](../monitoring-and-alerting/monitoring-and-alerting.md) 或安装你自己的 Prometheus operator 来使用 Prometheus 集成。Rancher 的 Istio chart 还默认安装 Kiali，确保你可以开箱即用地全面了解微服务。 + +Istio 已经脱离了使用 Helm 安装的方式，现在通过 Istioctl 二进制文件或 Istio Operator 进行安装。为了使用最简单的方式与 Istio 交互，Rancher 的 Istio 会维护一个 Helm Chart，该 Chart 使用 Istioctl 二进制文件来管理你的 Istio 安装。 + +此 Helm Chart 将在 UI 的**应用 & 市场**中提供。有权访问 Rancher Chart 应用商店的用户需要先设置 Istio，然后才能在项目中使用它。 + +## Istio 附带的工具 + +我们的 [Istio](https://istio.io/) 安装程序将 istioctl 二进制命令包装在一个 Helm chart 中，其中包括一个覆盖文件的选项，用来支持复杂的自定义配置。 + +它还包括以下内容： + +### Kiali + +Kiali 是一个全面的可视化辅助工具，用于绘制整个服务网格中的流量图。它允许你查看它们的连接方式，包括它们之间的流量速率和延迟。 + +你可以检查服务网格的运行状况，或深入查看单个组件的传入和传出请求。 + +### Jaeger + +Jaeger 是用于跟踪分布式系统的工具。我们的 Istio 安装程序包括能快速启动的一体化 [Jaeger](https://www.jaegertracing.io/) 安装。 + +请注意，这不是符合 Jaeger 生产要求的部署。此部署使用在内存中的存储组件，而 Jaeger 推荐在生产环境中使用持久存储组件。有关你所需的部署策略的更多信息，请参阅 [Jaeger 文档](https://www.jaegertracing.io/docs/latest/operator/#production-strategy)。 + +## 先决条件 + +在启用 Istio 之前，建议你先确认你的 Rancher worker 节点是否有足够的 [CPU 和内存](cpu-and-memory-allocations.md)来运行 Istio 的所有组件。 + +如果要在 RKE2 集群上安装 Istio，则需要执行一些额外的步骤。有关详细信息，请参阅[本节](#在-rke2-集群上安装-istio-的其他步骤) + +## 设置指南 + +如需了解如何设置 Istio 并在项目中使用它，请参阅[设置指南](../../how-to-guides/advanced-user-guides/istio-setup-guide/istio-setup-guide.md)。 + +## 卸载 Istio + +要从集群、命名空间或工作负载中删除 Istio 组件，请参阅[卸载 Istio](disable-istio.md)。 + +## 访问可视化 + +> 默认情况下，只有 cluster-admin 可以访问 Kiali。有关如何允许具有管理员、编辑或查看权限的角色访问它们的说明，请参阅[本节](rbac-for-istio.md)。 + +在集群中设置 Istio 后，你可以在 Rancher UI 中使用 Grafana、Prometheus 和 Kiali。 + +要访问 Grafana 和 Prometheus 可视化： + +1. 在左上角，单击 **☰ > 集群管理**。 +2. 在**集群**页面上，转到要可视化的集群，然后单击 **Explore**。 +3. 在左侧导航栏中，单击**监控**。 +4. 点击 **Grafana** 或任何其他仪表板。 + +要访问 Kiali 可视化： + +1. 在左上角，单击 **☰ > 集群管理**。 +2. 在**集群**页面上，转到要查看 Kiali 的集群，然后单击 **Explore**。 +3. 在左侧导航栏中，单击 **Istio**。 +4. 单击 **Kiali**。从这里，你可以访问**流量图**或**流量指标**选项卡，从而可视化网络指标。 + +默认情况下，prometheus 会拾取所有命名空间，并将数据用于 Kiali 图。如果你想使用不同的配置进行 prometheus 数据抓取，请参阅[选择器/抓取配置](configuration-options/selectors-and-scrape-configurations.md)。 + +你的角色决定了你对可视化的访问。只有 `cluster-admin` 角色可以使用 Grafana 和 Prometheus。默认情况下，只有 `cluster-admin` 可以使用 Kiali UI，但是 `cluster-admin` 可以通过编辑 Istio values.yaml 来允许其他角色进行访问。 + +## 架构 + +Istio 安装了一个服务网格，它使用 [Envoy](https://www.envoyproxy.io) Sidecar 代理来拦截到每个工作负载的流量。这些 sidecar 拦截并管理服务之间的通信，从而实现精细化观察并控制集群内的流量。 + +只有注入了 Istio sidecar 的工作负载可以通过 Istio 进行跟踪和控制。 + +如果命名空间启用了 Istio，部署到命名空间的新工作负载会自动具有 Istio sidecar。你需要为之前的工作负载手动启用 Istio。 + +有关 Istio sidecar 的更多信息，请参阅 [Istio sidecare-injection 文档](https://istio.io/docs/setup/kubernetes/additional-setup/sidecar-injection/)。有关 Istio 架构的更多信息，请参阅 [Istio 架构文档](https://istio.io/latest/docs/ops/deployment/architecture/)。 + +### 多个 Ingress + +默认情况下，每个 Rancher 配置的集群都有一个 NGINX Ingress Controller 来允许流量进入集群。Istio 还在 `istio-system` 命名空间中默认安装一个 Ingress Gateway。因此，你的集群将有两个 ingress。 + +![启用 Istio 的集群可以有两个 ingress，分别是默认的 Nginx ingress 和默认的 Istio controller](/img/istio-ingress.svg) + +可以通过[覆盖文件](configuration-options/configuration-options.md#覆盖文件)来启用其他 Istio Ingress Gateway。 + +### Egress 支持 + +默认情况下，Egress 网关是禁用的，但你可以在安装或升级时使用 values.yaml 或[覆盖文件](configuration-options/configuration-options.md#覆盖文件)启用它。 + +## 在 RKE2 集群上安装 Istio 的其他步骤 + +要在 RKE2 集群上安装 Istio，请按照[步骤](configuration-options/install-istio-on-rke2-cluster.md)进行操作。 + +## 在离线环境中升级 Istio + +现在，Istio Pod 安全策略默认启用。新值 `installer.releaseMirror.enabled` 已添加到 rancher-istio Chart 中，以启用和禁用支持离线升级的 Server。请注意，`installer.releaseMirror.enabled` 默认设置为 `false`。你可以在安装或升级时根据需要设置该值。按照以下步骤执行： + +1. 在 Rancher UI 中配置离线 Rancher 实例和离线自定义集群。 +2. 在集群中安装 Monitoring：**Cluster Explorer > Apps & Marketplace > Charts > Monitoring**。 +3. 将 Istio 所需的所有镜像拉入在离线环境中使用的私有镜像仓库。 +4. 在集群中安装 Istio：**Cluster Explorer > Apps & Marketplace > Charts > Istio**。 + +:::note + +你可以在新安装的 Istio 上启用 [Jaeger](https://www.jaegertracing.io/) 和 [Kiali](https://kiali.io/)。为确保 Jaeger 和 Kiali 正常工作，请在安装期间将 `values.yaml` 中的 `installer.releaseMirror.enabled` 设置为 `true`。 + +::: + +5. 升级 Istio。 + +:::caution + +如果你还没有执行操作，请设置 `installer.releaseMirror.enabled=true` 以升级 Istio。 + +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/kubernetes-distributions/kubernetes-distributions.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/kubernetes-distributions/kubernetes-distributions.md new file mode 100644 index 00000000000..b45cce6b4d7 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/kubernetes-distributions/kubernetes-distributions.md @@ -0,0 +1,34 @@ +--- +title: Kubernetes 发行版 +--- + + + + + +## K3s + +K3s 是一款轻量级、完全兼容的 Kubernetes 发行版，专为一系列用例设计，包括边缘计算、物联网、CI/CD、开发和将 Kubernetes 嵌入到应用程序中。它将系统打包为单个二进制文件，使用 sqlite3 作为默认存储，并提供用户友好的启动器，从而简化了 Kubernetes 的管理。K3s 包括 Local Storage 和负载均衡、Helm Chart 控制器和 Traefik CNI 等基本功能。它最大限度地减少了外部依赖，并提供了精简的 Kubernetes 体验。K3s 于 2020 年 6 月作为沙箱项目捐赠给 CNCF。 + +### K3s 与 Rancher + +- Rancher 允许在一系列平台上轻松配置 K3s，包括 Amazon EC2、DigitalOcean、Azure、vSphere 或现有服务器。 +- Kubernetes 集群的标准 Rancher 管理，包括所有概述[集群管理功能](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup#cluster-management-capabilities-by-cluster-type)。 + +## RKE2 + +RKE2 是 Rancher 开发的一款兼容 Kubernetes 的发行版。它是专门为美国联邦政府部门的安全性和合规性而设计的。 + +RKE2 的主要特性包括: + +1. **安全性和合规性重点**：RKE2 非常重视安全性和合规性，在“默认安全”框架下运行，适用于政府服务以及金融和医疗保健等高度监管的行业。 +1. **CIS Kubernetes Benchmark 一致性**：RKE2 经过预配置，符合 CIS Kubernetes Hardening Benchmark（目前支持 v1.23 和 v1.7），并且只需最少的手动干预。 +1. **FIPS 140-2 合规性**：RKE2 符合 FIPS 140-2 标准，使用经过 FIPS 验证的加密模块作为其组件。 +1. **嵌入式 ETCD**：RKE2 默认使用嵌入式 ETCD 作为其数据存储。这使其与标准的 Kubernetes 实践更加紧密地结合在一起，从而可以更好地与其他 Kubernetes 工具集成并降低错误配置的风险。 +1. **与上游 Kubernetes 保持一致**：RKE2 旨在与上游 Kubernetes 保持密切一致，降低在使用偏离标准 Kubernetes 实践的发行版时可能出现的不一致风险。 +1. **多重 CNI 支持**：RKE2 支持多个容器网络接口（CNI）插件，包括 Cilium、Calico 和 Multus。这对于具有各种生产设施的电信分发中心和工厂等用例至关重要。 + +## RKE2 与 Rancher + +- Rancher 允许在一系列平台上轻松配置 RKE2，包括 Amazon EC2、DigitalOcean、Azure、vSphere 或现有服务器。 +- Kubernetes 集群的标准 Rancher 管理，包括所有概述[集群管理功能](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup#cluster-management-capabilities-by-cluster-type)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/kubewarden/kubewarden.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/kubewarden/kubewarden.md new file mode 100644 index 00000000000..76f8c16d82a --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/kubewarden/kubewarden.md @@ -0,0 +1,31 @@ +--- +title: 使用 Kubewarden 进行高级策略管理 +--- + + + + + +Kubewarden 是一个策略引擎，用于保护和帮助管理集群资源。它允许通过策略验证和更改资源请求，包括上下文感知策略和验证镜像签名。它可以在监视或强制模式下运行策略，并提供集群状态的概述。 + +Kubewarden 旨在通过启用和简化“策略即代码”来成为通用策略引擎。Kubewarden 策略被编译到 WebAssembly 中： 它们体积小（400KB~2MB），沙箱式，安全且便携。它旨在通过迎合组织中的每个人来实现通用性： + +- 策略用户：使用 Kubernetes 自定义资源管理和声明策略，重新使用 Rego（OPA 和 Gatekeeper）中编写的现有策略。在 CI/CD 中测试群集外部的策略。 +- 策略开发人员：使用你喜欢的 Wasm 编译语言（Rego、Go、Rust、C#、Swift、Typescript 等）编写策略。重新使用你已经熟悉的工具、库和工作流的生态系统。 +- 策略分发器：策略是 OCI 工件，通过 OCI 仓库为它们提供服务，并在你的基础设施中使用行业标准，如 Software-Bill-Of-Materials 和工件签名。 +- 集群操作员：Kubewarden 是模块化的（OCI 注册表、PolicyServers、Audit Scanner、Controller）。配置你的部署以满足你的需求，隔离不同的租户。使用审核扫描程序和策略报告在集群中获取过去、当前和可能的违规行为的概述。 +- Kubewarden Integrator：将其用作编写新的 Kubewarden 模块和自定义策略的平台。 + +## Kubewarden 和 Rancher + +Kubewarden 的上游 Helm Chart 完全集成为 Rancher 应用程序，为安装选项提供了 UI。这些 Chart 还附带了尊重 Rancher 堆栈的默认设置（例如：不监管 Rancher 系统命名空间）以及默认的 PolicyServer 和策略。用户可以访问所有 Kubewarden 功能，并可以通过与 Kubernetes API 交互（例如：使用 kubectl）手动部署 PolicyServer 和策略。 + +Kubewarden 提供了对已删除的 Kubernetes Pod 安全策略的完全替换。 Kubewarden 还通过增强其安全功能，与最新版本的 Kubernetes 引入的新 Pod Security Admission 功能进行了集成。 + +## Kubewarden 和 Rancher Prime + +Kubewarden 的 Rancher UI 扩展将其集成到 Rancher UI 中。UI 扩展可自动安装和配置 Kubewarden 堆栈，并配置对 SUSE 维护的策略的访问。UI 扩展提供了对现成策略的策划目录的访问。使用 UI 扩展，人们可以浏览、安装和配置这些策略。 + +UI 扩展提供了 Kubewarden 堆栈组件及其行为的概述。这包括对 Kubewarden 指标和跟踪事件的访问。操作员可以了解策略对集群的影响并排查问题。 + +此外，UI 扩展还提供了 Policy Reporter UI，它可以直观地概述 Kubernetes 集群的合规性状态。通过此 UI，操作员可以快速识别所有不合规的 Kubernetes 资源，了解违规原因并采取相应行动。所有这一切都得益于 Rancher Prime 的支持。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/logging/custom-resource-configuration/custom-resource-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/logging/custom-resource-configuration/custom-resource-configuration.md new file mode 100644 index 00000000000..d719b72fe1d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/logging/custom-resource-configuration/custom-resource-configuration.md @@ -0,0 +1,12 @@ +--- +title: 自定义资源配置 +--- + + + + + +通过以下自定义资源配置 logging： + +- [Flow 和 ClusterFlow](flows-and-clusterflows.md) +- [Output 和 ClusterOutput](outputs-and-clusteroutputs.md) \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/logging/logging.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/logging/logging.md new file mode 100644 index 00000000000..dcc65014fbd --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/logging/logging.md @@ -0,0 +1,115 @@ +--- +title: Rancher Logging 集成 +description: Rancher 集成了主流的日志服务。了解集成日志服务的要求和优势，并在你的集群上启用 Logging。 +--- + + + + + +现在，Rancher 的日志管理由 [Logging operator](https://kube-logging.github.io/docs/) 提供支持，它取代了以前的内部解决方案。 + +## 启用 Logging + +你可以转到**应用**页面并安装 Logging 应用程序，从而为 Rancher 管理的集群启用 Logging： + +1. 转到要安装 Logging 的集群，然后单击 **Apps**。 +1. 点击 **Logging** 应用。 +1. 滚动到 Helm Chart README 的底部，然后单击**安装**。 + +**结果**：Logging 应用已部署到 `cattle-logging-system` 命名空间中。 + +## 卸载 Logging + +1. 转到要安装 Logging 的集群，然后单击 **Apps**。 +1. 点击**已安装的应用**。 +1. 转到 `cattle-logging-system` 命名空间并选中 `rancher-logging` 和 `rancher-logging-crd` 框。 +1. 单击**删除**。 +1. 确认**删除**。 + +**结果**：已卸载 `rancher-logging`。 + +## 架构 + +有关 Logging 应用程序工作原理的更多信息，请参阅[本节](logging-architecture.md)。 + +## RBAC + +Rancher Logging 有两个角色，分别是 `logging-admin` 和 `logging-view`。有关如何以及何时使用这些角色的更多信息，请参阅[此页面](rbac-for-logging.md)。 + +## 配置 Logging 自定义资源 + +要管理 `Flows`、`ClusterFlows`、`Outputs` 和 `ClusterOutputs`： + +1. 在左上角，单击 **☰ > 集群管理**。 +2. 在**集群**页面上，转到要配置 Logging 自定义资源的集群，然后单击 **Explore**。 +3. 在左侧导航栏中，单击 **Logging**。 + +### Flows 和 ClusterFlows + +有关配置 `Flows` 和 `ClusterFlows` 的帮助，请参阅[此页面](custom-resource-configuration/flows-and-clusterflows.md)。 + +### Outputs 和 ClusterOutputs + +有关配置 `Outputs` 和 `ClusterOutputs` 的帮助，请参阅[此页面](custom-resource-configuration/outputs-and-clusteroutputs.md)。 + +## 配置 Logging Helm Chart + +有关在安装或升级 Logging 应用程序时可配置的选项，请参阅[此页面](logging-helm-chart-options.md)。 + +### Windows 支持 + +你可以从 Windows 节点[启用 Logging](logging-helm-chart-options.md#启用禁用-windows-节点-logging)。 + +### 使用自定义 Docker 根目录 + +有关使用自定义 Docker 根目录的详细信息，请参阅[本节](logging-helm-chart-options.md#使用自定义-docker-根目录)。 + +### 处理污点和容忍度 + +有关如何在 Logging 应用程序中使用污点和容忍度的信息，请参阅[此页面](taints-and-tolerations.md)。 + +### 在 SELinux 上使用 Logging V2 + +有关在启用了 SELinux 的节点上使用 Logging 应用程序的信息，请参阅[本节](logging-helm-chart-options.md#启用-logging-应用程序以使用-selinux)。 + +### 其他日志来源 + +默认情况下，Rancher 会收集所有类型集群的 controlplane 组件和节点组件的日志。在某些情况下，也会收集其他日志。有关详细信息，请参阅[本节](logging-helm-chart-options.md#其他日志来源)。 + +## 故障排除 + +### 日志缓冲区导致 Pod 过载 + +根据你的配置，默认缓冲区大小可能太大并导致 Pod 故障。减少负载的一种方法是降低记录器的刷新间隔。这可以防止日志溢出缓冲区。你还可以添加更多刷新线程来处理大量日志试图同时填充缓冲区的情况。 + +有关如何配置日志缓冲区来满足企业需求的更完整说明，请参阅[缓冲区](https://kube-logging.github.io/docs/configuration/plugins/outputs/buffer/)和 [Fluentd 配置](https://kube-logging.github.io/docs/logging-infrastructure/fluentd/)的官方 Logging Operator 文档。 + +### `cattle-logging` 命名空间正在重新创建 + +如果你的集群之前在旧版 Rancher UI 的全局视图中部署了 Logging，`cattle-logging` 命名空间可能会不断被重新创建。 + +要解决这个问题，你可以将所有 `clusterloggings.management.cattle.io` 和 `projectloggings.management.cattle.io` 自定义资源从管理集群中针对该集群的命名空间中删除。 +这些自定义资源会导致 Rancher 在下游集群中创建 `cattle-logging` 命名空间（如果不存在）。 + +集群命名空间与集群 ID 匹配，因此我们需要找到每个集群的集群 ID。 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面上，转到要获取 ID 的集群，然后单击 **Explore**。 +2. 从以下其中一个 URL 中复制 `` 的内容。`` 是集群命名空间名称。 + +```bash +# Cluster Management UI +https:///c// + +# Cluster Dashboard +https:///dashboard/c// +``` + +现在我们有了 `` 命名空间，我们可以删除导致 `cattle-logging` 不断重新创建的自定义资源。 +*警告*：请确保当前未使用 Logging （从旧版 Rancher UI 全局视图中安装的版本）。 + +```bash +kubectl delete crd clusterloggings.management.cattle.io -n +kubectl delete crd projectloggings.management.cattle.io -n +``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/longhorn/longhorn.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/longhorn/longhorn.md new file mode 100644 index 00000000000..c3727f30ff8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/longhorn/longhorn.md @@ -0,0 +1,15 @@ +--- +title: 使用 Longhorn 进行云原生存储 +--- + + + + + +## Longhorn + +Longhorn 是官方的[云原生计算基金会（CNCF）](https://cncf.io/)项目，它为 Kubernetes 提供了一个可以在任何地方运行的强大的云原生分布式存储平台。当与 Rancher 相结合使用时，Longhorn 使你可以在 Kubernetes 环境中轻松、快速且可靠地部署高可用的持久块存储。 + +## Longhorn 与 Rancher + +凭借 Rancher Prime 和 Longhorn，用户可以通过 Rancher 应用商店一键轻松部署，并对托管集群进行生命周期管理；允许用户能够安装和升级，同时进行清空操作以实现优雅的操作。Longhorn 和 Rancher Prime 还提供了 Windows 的混合集群支持，Rancher 托管的镜像，通过 Rancher 的 UI 代理访问，以及使用 Longhorn 指标进行 Rancher 监控。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/longhorn/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/longhorn/overview.md new file mode 100644 index 00000000000..b5298c3c21d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/longhorn/overview.md @@ -0,0 +1,74 @@ +--- +title: 概述 +--- + + + + + +[Longhorn](https://longhorn.io/) 是一个轻量级、可靠且易于使用的 Kubernetes 分布式块存储系统。 + +Longhorn 是免费的开源软件。它最初由 Rancher Labs 开发，现在被开发为云原生计算基金会的沙箱项目。它可以通过 Helm、kubectl 或 Rancher UI 安装在任何 Kubernetes 集群上。有关其架构的更多信息，请参阅[此处](https://longhorn.io/docs/latest/concepts/)。 + +使用 Longhorn，你可以： + +- 使用 Longhorn 卷作为 Kubernetes 集群中分布式有状态应用程序的持久存储 +- 将块存储分区为 Longhorn 卷，以便你可以在有或没有云提供商的情况下使用 Kubernetes 卷 +- 跨多个节点和数据中心复制块存储以提高可用性 +- 将备份数据存储在 NFS 或 AWS S3 等外部存储中 +- 创建跨集群灾难恢复卷，以便使用另一个 Kubernetes 集群中的备份快速恢复主 Kubernetes 集群中的数据 +- 计划卷的定期快照，并将定期备份调度到 NFS 或兼容 S3 的辅助存储 +- 使用备份来恢复卷 +- 在不中断持久卷的情况下升级 Longhorn + +
Longhorn 仪表板
+ +![Longhorn 仪表板](/img/longhorn-screenshot.png) + +### 使用 Rancher 安装 Longhorn + +1. 满足所有[安装要求](https://longhorn.io/docs/latest/deploy/install/#installation-requirements)。 +1. 转到要安装 Longhorn 的集群。 +1. 单击 **Apps**。 +1. 单击 **Charts**。 +1. 单击 **Longhorn**。 +1. 可选：要自定义初始设置，请单击 **Longhorn 默认设置**并编辑配置。如需自定义设置的帮助，请参阅 [Longhorn 文档](https://longhorn.io/docs/latest/references/settings/)。 +1. 单击**安装**。 + +**结果**：Longhorn 已部署到 Kubernetes 集群中。 + +### 从 Rancher UI 访问 Longhorn + +1. 转到安装了 Longhorn 的集群。在左侧导航菜单中，单击 **Longhorn**。 +1. 在此页面上，你可以编辑 Longhorn 管理的 Kubernetes 资源。要查看 Longhorn UI，请单击**概述**中的 **Longhorn** 按钮。 + +**结果**：你将转到 Longhorn UI，在这里你可以管理 Kubernetes 集群中的 Longhorn 卷及其副本，以及可能存在于另一个 Kubernetes 集群或 S3 中的 Longhorn 存储辅助备份。 + +### 从 Rancher UI 卸载 Longhorn + +1. 转到安装了 Longhorn 的集群，然后单击 **Apps**。 +1. 点击**已安装的应用**。 +1. 转到 `longhorn-system` 命名空间并选中 `longhorn` 和 `longhorn-crd` 应用程序旁边的框。 +1. 单击**删除**并确认**删除**。 + +**结果**：Longhorn 已被卸载。 + +### GitHub 仓库 + +Longhorn 项目可在[此处](https://github.com/longhorn/longhorn)获取。 + +### 文档 + +Longhorn 文档在[此处](https://longhorn.io/docs/)。 + +### 架构 + +Longhorn 为每个卷创建专用的存储控制器，并在多个节点上存储的多个副本之间同步复制该卷。 + +存储控制器和副本本身是使用 Kubernetes 编排的。 + +有关其架构的更多信息，请参阅[此处](https://longhorn.io/docs/latest/concepts/)。 + +
Longhorn 架构
+ +![Longhorn 架构](/img/longhorn-architecture.svg) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/monitoring-and-alerting/built-in-dashboards.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/monitoring-and-alerting/built-in-dashboards.md index 0161de45166..5dd50402e17 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/monitoring-and-alerting/built-in-dashboards.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/monitoring-and-alerting/built-in-dashboards.md @@ -110,4 +110,8 @@ Alertmanager UI 可让你查看最近触发的告警。 ![PrometheusRules UI](/img/prometheus-rules-ui.png) -有关在 Rancher 中配置 PrometheusRule 的更多信息，请参阅[此页面](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md)。 \ No newline at end of file +有关在 Rancher 中配置 PrometheusRule 的更多信息，请参阅[此页面](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md)。 + +## 旧版 UI + +有关在引入 `rancher-monitoring` 应用程序之前 Rancher v2.2 到 v2.4 中可用仪表板的信息，请参阅 [Rancher v2.0—v2.4 文档](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/cluster-monitoring/viewing-metrics.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md new file mode 100644 index 00000000000..cebb2e4323f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md @@ -0,0 +1,106 @@ +--- +title: 监控和告警 +description: Prometheus 允许你查看来自不同 Rancher 和 Kubernetes 对象的指标。了解监控范围以及如何启用集群监控 +--- + + + + + +你可以使用 `rancher-monitoring` 应用，将业界领先的开源监控和告警解决方案快速部署到你的集群中。 + +在 Rancher v2.5 中引入的 `rancher-monitoring` operator 由 [Prometheus](https://prometheus.io/)、[Grafana](https://grafana.com/grafana/)、[Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)， [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) 和 [Prometheus adapter](https://github.com/DirectXMan12/k8s-prometheus-adapter) 提供支持。 + +有关在 Rancher v2.2 到 v2.4 中可用的 V1 监控和告警的信息，请参阅 Rancher v2.0 到 v2.4 文档中的[集群监控](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/cluster-monitoring/cluster-monitoring.md)，[告警](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/cluster-alerts/cluster-alerts.md)，[通知](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/notifiers.md)和[工具](/versioned_docs/version-2.0-2.4/reference-guides/rancher-project-tools/rancher-project-tools.md)。 + +使用 `rancher-monitoring` 应用程序，你可以快速部署领先的开源监控和告警解决方案到你的集群上。 + +### 功能 + +Prometheus 支持查看 Rancher 和 Kubernetes 对象的指标。通过使用时间戳，Prometheus 能让你通过 Rancher UI 或 Grafana（与 Prometheus 一起部署的分析查看平台）以更容易阅读的图表和视觉形式来查询和查看这些指标。 + +通过查看 Prometheus 从集群的 controlplane、节点和 deployment 中抓取的数据，你可以随时了解集群中发生的所有事件。然后，你可以使用这些分析来更好地运行你的环境，例如在系统紧急情况发生之前阻止它们、制定维护策略，或恢复崩溃的服务器。 + +Monitoring 应用： + +- 监控集群节点、Kubernetes 组件和软件部署的状态和进程。 +- 根据 Prometheus 收集的指标定义告警。 +- 创建自定义 Grafana 仪表板。 +- 使用 Prometheus Alertmanager 通过电子邮件、Slack、PagerDuty 等配置告警通知。 +- 根据 Prometheus 收集的指标，将预先计算的、经常需要的，或计算成本高的表达式定义为新的时间序列。 +- 通过 Prometheus Adapter，将从 Prometheus 收集的指标公开给 Kubernetes Custom Metrics API，以便在 HPA 中使用。 + +有关监控组件如何协同工作的说明，请参阅 [Monitoring 工作原理](how-monitoring-works.md)。 + +## 默认组件和部署 + +### 内置仪表板 + +默认情况下，监控应用将 Grafana 仪表板（由 [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) 项目策划）部署到集群上。 + +它还部署一个 Alertmanager UI 和一个 Prometheus UI。有关这些工具的更多信息，请参见[内置仪表板](built-in-dashboards.md)。 + +### 默认指标 Exporter + +默认情况下，Rancher Monitoring 会部署 Exporter（例如 [node-exporter](https://github.com/prometheus/node_exporter) 和 [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)）。 + +这些默认 Exporter 会自动从 Kubernetes 集群的所有组件（包括工作负载）中抓取 CPU 和内存的指标。 + +### 默认告警 + +Monitoring 应用会默认部署一些告警。要查看默认告警，请转到 [Alertmanager UI](built-in-dashboards.md#alertmanager-ui) 并单击**展开所有组**。 + +### Rancher UI 中公开的组件 + +有关 Rancher UI 中公开的监控组件列表，以及编辑它们的常见用例，请参阅[本节](how-monitoring-works.md#rancher-ui-中公开的组件)。 + +## RBAC + +有关配置 monitoring 访问权限的信息，请参阅[此页面](rbac-for-monitoring.md)。 + +:::note + +Rancher 和 Project 的读取权限并不一定适用于监控相关资源. 查看 [monitoring-ui-view](rbac-for-monitoring.md#其他监控角色) 获取更多详细信息. + +::: + +## 指南 + +- [启用 monitoring](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/enable-monitoring.md) +- [卸载 monitoring](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/uninstall-monitoring.md) +- [Monitoring 工作负载](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/set-up-monitoring-for-workloads.md) +- [自定义 Grafana 仪表板](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/customize-grafana-dashboard.md) +- [持久化 Grafana 仪表板](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/create-persistent-grafana-dashboard.md) +- [调试高内存使用率](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/debug-high-memory-usage.md) + +## 配置 + +### 在 Rancher 中配置 Monitoring 资源 + +此处的配置参考假设你已经熟悉 monitoring 组件的协同工作方式。如需更多信息，请参阅 [monitoring 的工作原理](how-monitoring-works.md)。 + +- [ServiceMonitor 和 PodMonitor](../../reference-guides/monitoring-v2-configuration/servicemonitors-and-podmonitors.md) +- [接收器](../../reference-guides/monitoring-v2-configuration/receivers.md) +- [路由](../../reference-guides/monitoring-v2-configuration/routes.md) +- [PrometheusRule](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheusrules.md) +- [Prometheus](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/prometheus.md) +- [Alertmanager](../../how-to-guides/advanced-user-guides/monitoring-v2-configuration-guides/advanced-configuration/alertmanager.md) + +### 配置 Helm Chart 选项 + +有关 `rancher-monitoring` Chart 选项的更多信息，包括设置资源限制和请求的选项，请参阅 [Helm Chart 选项](../../reference-guides/monitoring-v2-configuration/helm-chart-options.md)。 + +## Windows 集群支持 + +如果 Monitoring 部署到 RKE1 Windows 集群，Monitoring V2 将自动部署 [windows-exporter](https://github.com/prometheus-community/windows_exporter) DaemonSet 并设置 ServiceMonitor，以从每个部署的 Pod 中收集指标。这将使用 `windows_` 指标填充 Prometheus，这些指标与 [node_exporter](https://github.com/prometheus/node_exporter) 为 Linux 主机导出的 `node_` 指标类似。 + +为了能够为 Windows 完全部署 Monitoring V2，你的所有 Windows 主机都必须至少具有 v0.1.0 的 [wins](https://github.com/rancher/wins) 版本。 + +有关如何在现有 Windows 主机上升级 wins 版本的更多信息，请参阅 [Windows 集群对 Monitoring V2 的支持](windows-support.md)。 + + +## 已知问题 + +有一个[已知问题](https://github.com/rancher/rancher/issues/28787#issuecomment-693611821)，即 K3s 集群需要的内存超过分配的默认内存。如果你在 K3s 集群上启用 Monitoring，将 `prometheus.prometheusSpec.resources.memory.limit` 设置为 2500 Mi，并将 `prometheus.prometheusSpec.resources.memory.request` 设置为 1750 Mi。 + +如需获取意见和建议，请参阅[调试高内存使用情况](../../how-to-guides/advanced-user-guides/monitoring-alerting-guides/debug-high-memory-usage.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/neuvector/neuvector.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/neuvector/neuvector.md new file mode 100644 index 00000000000..d2f1257309f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/neuvector/neuvector.md @@ -0,0 +1,27 @@ +--- +title: 使用 NeuVector 实现容器安全 +--- + + + + + +NeuVector 是唯一一个100%开源、零信任的容器安全平台。在整个容器生命周期中持续扫描。清除安全路障。从一开始就制定安全策略，以最大限度地提高开发人员的灵活性。NeuVector 提供从构建到生产的漏洞和合规性扫描与管理。独特的 NeuVector 运行时保护通过七层容器防火墙保护集群内的网络连接以及集群的入口/出口。此外，NeuVector 还监视容器和主机上的进程和文件活动，以阻止未经授权的活动。 + +## NeuVector 和 Rancher + +所有 NeuVector 功能均可通过 Rancher 进行集成部署并单点登录 NeuVector 控制台。Rancher 集群管理员能够在其集群上部署和管理 NeuVector 部署，Helm 值、configMaps、自定义资源定义（CRD）和 NeuVector 控制台轻松配置 NeuVector。 + +使用 NeuVector 和 Rancher： + +- 部署、管理和保护多个集群。 +- 管理和报告 Rancher 工作负载和节点的漏洞和合规性结果。 + +## NeuVector Prime 和 Rancher Prime + +Rancher Manager 的 NeuVector UI 扩展可用于并支持 Rancher Prime 和 NeuVector Prime 客户。此扩展提供： + +- NeuVector 自动化部署，包括 Rancher Prime NeuVector 扩展仪表板。 +- 访问每个集群的重要安全信息，如关键安全事件、漏洞扫描结果和入口/出口暴露。 +- 直接对 Rancher 资源（例如节点和容器/ Pod）进行集成漏洞 (CVE) 和合规扫描。 +- 集成操作，如手动触发 Rancher 资源的扫描。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/neuvector/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/neuvector/overview.md new file mode 100644 index 00000000000..2a23c695335 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/neuvector/overview.md @@ -0,0 +1,196 @@ +--- +title: 概述 +--- + + + + + +### Rancher 中的 NeuVector 集成 + +[NeuVector 5.x](https://open-docs.neuvector.com/) 是一个开源的，以容器为中心的安全应用程序，Rancher 已集成 NeuVector。NeuVector 在运行时为关键应用程序和数据提供实时的合规、可见和保护功能。NeuVector 提供具有 CIS Benchmark 和漏洞扫描的防火墙、容器进程/文件系统监控和安全审计。有关 Rancher 安全性的更多信息，请参阅[安全文档](../../reference-guides/rancher-security)。 + +NeuVector 可以通过 Helm Chart 启用。你可以在 **Apps** 或 Rancher UI 中的 **Cluster Tools** 中安装该 Chart。安装 Helm Chart 后，用户可以轻松地[在 Rancher 中部署和管理 NeuVector 集群](https://open-docs.neuvector.com/deploying/rancher#deploy-and-manage-neuvector-through-rancher-apps-marketplace)。 + +### 使用 Rancher 安装 NeuVector + +Harvester Helm Chart 用于管理 Rancher 中 NeuVector UI 的访问，用户可以在 Rancher 中直接跳转，然后部署和管理 NeuVector 集群。 + +**通过 Apps 导航并安装 NeuVector Chart：** + +1. 单击 **☰ > 集群管理**。 +1. 在 Clusters 页面上，转到要部署 NeuVector 的集群，然后单击 **Explore**。 +1. 转到 **Apps > Charts**，然后从 Chart 仓库中安装 **NeuVector**。 +1. 不同的集群类型需要不同的容器运行时。配置 Helm Chart 值时，转到**容器运行时**，然后根据集群类型选择运行时。最后，再次单击**安装**。 + +以下是一些例子： + +- RKE1：`docker` +- K3s 和 RKE2：`k3scontainerd` +- AKS：`containerd` 适用于 v1.19 及更高版本 +- EKS：`docker` 适用于 v1.22 及以下版本；`containerd` 适用于 v1.23 及更高版本 +- GKE：`containerd` (请参阅 [Google 文档](https://cloud.google.com/kubernetes-engine/docs/concepts/using-containerd)了解更多信息) + + :::note + + 在安装过程中一次只能选择一个容器运行时引擎。 + + ::: + +**通过集群工具导航并安装 NeuVector Chart：** + +1. 单击 **☰ > 集群管理**。 +1. 在 Clusters 页面上，转到要部署 NeuVector 的集群，然后单击 **Explore**。 +1. 点击左侧导航栏底部的**集群工具**。 +1. 按照上面的步骤 4 相应地选择你的容器运行时，然后再次单击**安装**。 + +### 从 Rancher UI 访问 NeuVector + +1. 导航到安装了 NeuVector 的集群的 Cluster Explorer。在左侧导航栏中，单击 **NeuVector**。 +1. 单击外部链接以转到 NeuVector UI。选择链接后，用户必须接受`最终用户许可协议`才能访问 NeuVector UI。 + +### 从 Rancher UI 卸载 NeuVector + +**通过 Apps 卸载：** + +1. 单击 **☰ > 集群管理**。 +1. 在 **Apps** 下，点击 **Installed Apps**。 +1. 在 `cattle-neuvector-system` 下，选择 NeuVector 应用程序（如果需要，还可以选择相关的 CRD），然后单击**删除**。 + +**通过集群工具卸载：** + +1. 单击 **☰ > 集群管理**。 +1. 单击屏幕左下角的**集群工具**，然后单击 NeuVector Chart 下方的垃圾桶图标。如果需要，选择`删除与此应用关联的 CRD`，然后单击**删除**。 + +### GitHub 仓库 + +NeuVector 项目在[这里](https://github.com/neuvector/neuvector)。 + +### 文档 + +NeuVector 文档在[这里](https://open-docs.neuvector.com/)。 + +### 架构 + +NeuVector 安全解决方案包含四种类型的安全容器，分别是 Controller、Enforcer、Manager 和 Scanner。它还提供了一个称为 All-in-One 的特殊容器（主要用于 Docker 原生部署），能将 Controller、Enforcer 和 Manager 功能组合在一个容器中。此外，还有一个 Updater，运行该程序时会更新 CVE 数据库。 + +- **Controller：** 管理 NeuVector Enforcer 容器；为管理控制台提供 REST API。 +- **Enforcer：** 执行安全策略。 +- **Manager：** 提供一个 web-UI 和 CLI 控制台来管理 NeuVector 平台。 +- **All-in-One：** 包括 Controller、Enforcer 和 Manager。 +- **Scanner：** 对镜像、容器和节点执行漏洞和合规性扫描。 +- **Updater：** 更新 Neuvector 的 CVE 数据库（运行的时候）；重新部署 scanner pod。 + +
**NeuVector 安全容器：**
+ +![NeuVector 安全容器](/img/neuvector-security-containers.png) + +
**NeuVector 架构：**
+ +![NeuVector 架构](/img/neuvector-architecture.png) + +要了解有关 NeuVector 架构的更多信息，请参阅[此处](https://open-docs.neuvector.com/basics/overview#architecture)。 + +### CPU 和内存分配 + +以下是默认 NeuVector Chart 安装部署的最低计算资源推荐。请注意，未设置资源限制。 + +| 容器 | CPU - 请求 | 内存 - 请求 | +| ---------- | ----------------------------- | ----------- | +| Controller | 3（每个控制器需要 1GB 1vCPU） | \* | +| Enforcer | 所有节点上 (500MB .5vCPU) | 1GB | +| Manager | 1 (500MB .5vCPU) | \* | +| Scanner | 3 (100MB .5vCPU) | \* | + +\* Controller、Manager 和 Scanner 容器合计至少需要 1GB 内存。 + +### 强化集群支持 - Calico 和 Canal + + + + +- 如果 PSP 设置为 true，则所有 NeuVector 组件都是可部署的。 + +你需要为强化集群环境进行额外的配置，如下所示： + +1. 单击 **☰ > 集群管理**。 +1. 选择你创建的集群，并点击 **Explore**。 +1. 在左侧导航栏中，点击 **Apps**。 +1. 安装（或升级到）NeuVector 版本 `100.0.1+up2.2.2`。 + +- 在**编辑选项** > **其它配置**下，选中复选框来启用 **Pod 安全策略**（无需其他配置）： + + ![为 RKE1 强化集群启用 PSP](/img/psp-nv-rke.png) + +1. 点击右下角的**安装**。 + + + + +- 如果 PSP 设置为 true，则可以部署 NeuVector 组件 Controller 和 Enforcer。 + +**仅适用于 NeuVector Chart 版本 100.0.0+up2.2.0：** + +- 对于 Manager、Scanner 和 Updater 组件，需要进行额外的配置，如下所示： + +``` +kubectl patch deploy neuvector-manager-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}' +kubectl patch deploy neuvector-scanner-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}' +kubectl patch cronjob neuvector-updater-pod -n cattle-neuvector-system --patch '{"spec":{"jobTemplate":{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}}}' +``` + +
+ +你需要为强化集群环境进行额外的配置。 + +> **注意：** 你必须更新 RKE2 和 K3s 强化集群中的配置，如下所示。 + +1. 单击 **☰ > 集群管理**。 +1. 选择你创建的集群，并点击 **Explore**。 +1. 在左侧导航栏中，点击 **Apps**。 +1. 安装（或升级到）NeuVector 版本 `100.0.1+up2.2.2`。 + +- 在**编辑选项** > **其它配置**下，选中复选框来启用 **Pod 安全策略**。请注意，对于 `Manager runAsUser ID`，`Scanner runAsUser ID` 和 `Updater runAsUser ID`，你还必须输入大于 `0` 的值： + + ![为 RKE2 和 K3s 强化集群启用 PSP](/img/psp-nv-rke2.png) + +1. 点击右下角的**安装**。 + + + + +### 启用 SELinux 的集群支持 - Calico 和 Canal + +要在 RKE2 集群上启用 SELinux，请执行以下步骤： + +- 如果 PSP 设置为 true，则可以部署 NeuVector 组件 Controller 和 Enforcer。 + +**仅适用于 NeuVector Chart 版本 100.0.0+up2.2.0：** + +- 对于 Manager、Scanner 和 Updater 组件，需要进行额外的配置，如下所示： + +``` +kubectl patch deploy neuvector-manager-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}' +kubectl patch deploy neuvector-scanner-pod -n cattle-neuvector-system --patch '{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}' +kubectl patch cronjob neuvector-updater-pod -n cattle-neuvector-system --patch '{"spec":{"jobTemplate":{"spec":{"template":{"spec":{"securityContext":{"runAsUser": 5400}}}}}}}' +``` + +### 离线环境中的集群支持 + +- 所有 NeuVector 组件都可部署在离线环境中的集群上，无需任何额外配置。 + +### 支持限制 + +- 目前仅支持管理员和集群所有者。 + +- 不支持 Fleet 多集群部署。 + +- Windows 集群不支持 NeuVector。 + +### 其他限制 + +- 目前，如果 NeuVector partner Chart 已存在，则 NeuVector 功能 Chart 的安装会失败。要解决此问题，请卸载 NeuVector partner Chart 并重新安装 NeuVector 功能 Chart。 + +- Controller 未准备好时，有可能无法从 Rancher UI 访问 NeuVector UI。在此期间，Controller 将尝试重新启动，并且需要几分钟才能进入 active 状态。 + +- 安装 NeuVector Chart 时，不会针对不同的集群类型自动检测容器运行时。要解决此问题，你可以手动指定运行时。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/rancher-desktop.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/rancher-desktop.md new file mode 100644 index 00000000000..5e418069595 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/integrations-in-rancher/rancher-desktop.md @@ -0,0 +1,33 @@ +--- +title: 桌面上的 Kubernetes 与 Rancher Desktop +--- + + + + + +Rancher Desktop 将用于桌面开发和测试云原生应用程序的基本工具捆绑在一起。 + +如果你在本地计算机上运行适用于云环境的应用程序，则通常需要进行大量准备工作。你需要选择一个容器运行时,安装 Kubernetes 和流行的实用程序，并且可能还需要设置虚拟机。单独安装组件并使它们协同工作可能是一个耗时的过程。 + +为了降低复杂性，Rancher Desktop 为团队提供了以下主要功能： + +- 在 macOS、Linux 和 Windows 操作系统上进行简单易用的安装。 +- K3s，一个易于使用的轻量级 Kubernetes 发行版。 +- 能够在 Kubernetes 版本之间轻松切换。 +- 由 Rancher 支持的基于 GUI 的集群仪表板，用于浏览本地集群。 +- 自由选择容器引擎：dockerd（moby）或 containerd。 +- 用于配置应用程序以满足你的需求的首选项设置。 +- 容器、基于 Kubernetes 的开发和操作工作流程所需的捆绑工具。 +- 定期更新，使捆绑工具保持最新。 +- 与流行的工具/IDE 集成，包括 VS Code 和 Skaffold。 +- 镜像和注册表访问控制。 +- 支持 Docker 扩展。 + +访问 [Rancher Desktop](https://rancherdesktop.io) 网站并阅读[文档](https://docs.rancherdesktop.io/)以了解更多信息。 + +要在你的机器上安装 Rancher Desktop，请参阅[安装指南](https://docs.rancherdesktop.io/getting-started/installation)。 + +## 在 Rancher Desktop 上尝试 Rancher + +Rancher Desktop 提供了轻松试用容器化、基于 Helm 的应用程序所需的设置和工具。你可以按照此[操作指南](https://docs.rancherdesktop.io/how-to-guides/rancher-on-rancher-desktop)，使用 Rancher Desktop 开始使用 Rancher Kubernetes 管理平台。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/about-the-api/about-the-api.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/about-the-api/about-the-api.md new file mode 100644 index 00000000000..f1756eb6e5d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/about-the-api/about-the-api.md @@ -0,0 +1,93 @@ +--- +title: API +--- + + + + + +## 如何使用 API + +API 有自己的用户界面，你可以从 Web 浏览器访问它。这是查看资源、执行操作以及查看等效 cURL 或 HTTP 请求和响应的一种简单的方法。要访问它： + + + + +1. 单击右上角的用户头像。 +1. 单击**账号 & API 密钥**。 +1. 在 **API 密钥**下，找到 **API 端点**字段并单击链接。该链接类似于 `https:///v3`，其中 `` 是 Rancher deployment 的完全限定域名。 + + + + +转到位于 `https:///v3` 的 URL 端点，其中 `` 是你的 Rancher deployment 的完全限定域名。 + + + + +## 认证 + +API 请求必须包含认证信息。认证是通过 [API 密钥](../user-settings/api-keys.md)使用 HTTP 基本认证完成的。API 密钥可以创建新集群并通过 `/v3/clusters/` 访问多个集群。[集群和项目角色](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md)会应用于这些键，并限制账号可以查看的集群和项目以及可以执行的操作。 + +默认情况下，某些集群级别的 API 令牌是使用无限期 TTL（`ttl=0`）生成的。换言之，除非你让令牌失效，否则 `ttl=0` 的 API 令牌永远不会过期。有关如何使 API 令牌失效的详细信息，请参阅 [API 令牌](api-tokens.md)。 + +## 发出请求 + +该 API 通常是 RESTful 的，但是还具有多种功能。这些功能可以使客户端发现所有内容，因此可以编写通用客户端，而不必为每种资源编写特定代码。有关通用 API 规范的详细信息，请参阅[此处](https://github.com/rancher/api-spec/blob/master/specification.md)。 + +- 每种类型都有一个 Schema，这个 Schema 描述了以下内容： + - 用于获取此类资源集合的 URL + - 资源可以具有的每个字段及其类型、基本验证规则、是必填还是可选字段等 + - 在此类资源上可以执行的每个操作，以及它们的输入和输出（也作为 schema） + - 允许过滤的每个字段 + - 集合本身或集合中的单个资源可以使用的 HTTP 操作方法 + + +- 因此，你可以只加载 schema 列表并了解 API 的所有信息。实际上，这是 API 的 UI 工作方式，它不包含特定于 Rancher 本身的代码。每个 HTTP 响应中的 `X-Api-Schemas` 标头都会发送获取 Schemas 的 URL。你可以按照每个 schema 上的 `collection` 链接了解要在哪里列出资源，并在返回资源中的其他 `links` 中获取其他信息。 + +- 在实践中，你可能只想构造 URL 字符串。我们强烈建议将此限制为在顶层列出的集合 (`/v3/`)，或获取特定资源 (`/v3//`)。除此之外的任何内容都可能在将来的版本中发生更改。 + +- 资源之间相互之间有联系，称为链接（links）。每个资源都包含一个 `links` 映射，其中包含链接名称和用于检索该信息的 URL。同样，你应该 `GET` 资源并遵循 `links` 映射中的 URL，而不是自己构造这些字符串。 + +- 大多数资源都有操作（action），表示可以执行某个操作或改变资源的状态。要使用操作，请将 HTTP `POST` 请求发送到 `actions` 映射中你想要的操作的 URL。某些操作需要输入或生成输出，请参阅每种类型的独立文档或 schema 以获取具体信息。 + +- 要编辑资源，请将 HTTP `PUT` 请求发送到资源上的 `links.update` 链接，其中包含要更改的字段。如果链接丢失，则你无权更新资源。未知字段和不可编辑的字段将被忽略。 + +- 要删除资源，请将 HTTP `DELETE` 请求发送到资源上的 `links.remove` 链接。如果链接丢失，则你无权更新资源。 + +- 要创建新资源，HTTP `POST` 到 schema（即 `/v3/`）中的集合 URL。 + +## 过滤 + +你可以使用 HTTP 查询参数的公共字段在服务器端过滤大多数集合。`filters` 映射显示了可以过滤的字段，以及过滤后的值在你发起的请求中是什么。API UI 具有设置过滤和显示适当请求的控件。对于简单的 "equals" 匹配，它只是 `field=value`。你可以将修饰符添加到字段名称，例如 `field_gt=42` 表示“字段大于 42”。详情请参阅 [API 规范](https://github.com/rancher/api-spec/blob/master/specification.md#filtering)。 + +## 排序 + +你可以使用 HTTP 查询参数的公共字段在服务器端排序大多数集合。`sortLinks` 映射显示了可用的排序，以及用于获取遵循该排序的集合的 URL。它还包括当前响排序依据的信息（如果指定）。 + +## 分页 + +默认情况下，API 响应以每页 100 个资源的限制进行分页。你可以通过 `limit` 查询参数进行更改，最大为 1000，例如 `/v3/pods?limit=1000`。集合响应中的 `pagination` 映射能让你知道你是否拥有完整的结果集，如果没有，则会指向下一页的链接。 + +## 捕获 Rancher API 调用 + +你可以使用浏览器开发人员工具来捕获 Rancher API 的调用方式。例如，你可以按照以下步骤使用 Chrome 开发人员工具来获取用于配置 RKE 集群的 API 调用： + +1. 在 Rancher UI 中，转到**集群管理**并单击**创建**。 +1. 单击某个集群类型。此示例使用 Digital Ocean。 +1. 使用集群名称和节点模板填写表单，但不要单击**创建**。 +1. 在创建集群之前，你需要打开开发人员工具才能看到正在记录的 API 调用。要打开工具，右键单击 Rancher UI，然后单击**检查**。 +1. 在开发者工具中，单击 **Network** 选项卡。 +1. 在 **Network** 选项卡上，确保选择了 **Fetch/XHR**。 +1. 在 Rancher UI 中，单击**创建**。在开发者工具中，你应该会看到一个名为 `cluster?_replace=true` 的新网络请求。 +1. 右键单击 `cluster?_replace=true` 并单击**复制 > 复制为 cURL**。 +1. 将结果粘贴到文本编辑器中。你将能够看到 POST 请求，包括被发送到的 URL、所有标头以及请求的完整正文。此命令可用于从命令行创建集群。请注意，请求包含凭证，因此请将请求存储在安全的地方。 + +### 启用在 API 中查看 + +你还可以查看针对各自集群和资源捕获的 Rancher API 调用。 默认情况下不启用此功能。 要启用它： + +1. 单击 UI 右上角的 **用户图标**，然后从下拉菜单中选择 **偏好设置** +1. 在**高级功能**部分下，单击**启用"在 API 中查看"** + +选中后，**在 API 中查看**链接现在将显示在 UI 资源页面上的 **⋮** 子菜单下。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/backup-restore-configuration/backup-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/backup-restore-configuration/backup-configuration.md index 24505753d86..005e93110aa 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/backup-restore-configuration/backup-configuration.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/backup-restore-configuration/backup-configuration.md @@ -68,6 +68,12 @@ kubectl create secret generic encryptionconfig \ ### S3 +:::caution + +如果你使用 S3 备份目标，请确保每个集群都有自己的存储桶或文件夹。Rancher 将使用集群配置的 S3 存储桶或文件夹中的可用快照来填充快照信息。 + +::: + S3 存储位置包含以下配置字段： 1. **凭证密文**（可选）：如果你需要使用 AWS 访问密钥（access key）和密文密钥（secret key）来访问 S3 存储桶，请使用带有密钥和指令 `accessKey` 和 `secretKey` 的凭证来创建密文。它可以是任意一个命名空间。你可以点击[此处](#credentialsecret-示例)查看示例密文。如果运行 operator 的节点在 EC2 中，并且设置了允许它们访问 S3 的 IAM 权限，则此指令是不必要的（如[本节](#ec2-节点访问-s3-的-iam-权限)所述）。凭证密文下拉菜单列出了所有命名空间的密文。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/backup-restore-configuration/backup-restore-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/backup-restore-configuration/backup-restore-configuration.md new file mode 100644 index 00000000000..6fbc1f202af --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/backup-restore-configuration/backup-restore-configuration.md @@ -0,0 +1,12 @@ +--- +title: Rancher 备份配置参考 +--- + + + + + +- [备份配置](backup-configuration.md) +- [还原配置](restore-configuration.md) +- [存储位置配置](storage-configuration.md) +- [Backup 和 Restore 自定义资源示例](examples.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/best-practices.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/best-practices.md new file mode 100644 index 00000000000..11f4ee82e41 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/best-practices.md @@ -0,0 +1,21 @@ +--- +title: 最佳实践 +--- + + + + + +本节介绍 Rancher 实现的最佳实践，其中包括对 Kubernetes、Docker、容器等技术的使用建议。最佳实践旨在利用 Rancher 及其客户的运营经验，助你更好地实现 Rancher。 + +如果你对用例的实际应用有任何疑问，请联系客户成功经理或支持中心。 + +你可以在左侧导航栏快速找到管理和部署 Rancher Server 的最佳实践。 + +如需查看更多最佳实践指南，请参见： + +- [安全类文档](../rancher-security/rancher-security.md) +- [Rancher 博客](https://www.suse.com/c/rancherblog/) +- [Rancher 论坛](https://forums.rancher.com/) +- [Rancher 用户的 Slack 群组](https://slack.rancher.io/) +- [B 站](https://space.bilibili.com/430496045/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/monitoring-best-practices.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/monitoring-best-practices.md index d880b03bd0b..c48a1d04995 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/monitoring-best-practices.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/monitoring-best-practices.md @@ -86,7 +86,7 @@ Prometheus 不是用于长期存储指标的，它只用于短期存储。 如果你有一个（微）服务架构，在该架构中集群的多个单独的工作负载相互通信，那么拥有这些流量的详细指标和跟踪是非常重要的，因为这可以帮助你了解所有这些工作负载之间的通信方式，以及问题或瓶颈可能出现的地方。 -当然，你可以监控所有工作负载中的所有内部流量，并将这些指标暴露给 Prometheus，但这相当耗费精力。像 Istio 这样的服务网格（可以通过[单击](https://rancher.com/docs/rancher/v2.6/en/istio/)在 Rancher 中安装）可以自动完成这项工作，并提供所有 Service 之间流量的丰富的遥测数据。 +当然，你可以监控所有工作负载中的所有内部流量，并将这些指标暴露给 Prometheus，但这相当耗费精力。像 Istio 这样的服务网格（可以通过[单击](../../../pages-for-subheaders/istio.md)在 Rancher 中安装）可以自动完成这项工作，并提供所有 Service 之间流量的丰富的遥测数据。 ## 真实用户监控 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md new file mode 100644 index 00000000000..e12f1db2498 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md @@ -0,0 +1,23 @@ +--- +title: Rancher 管理集群的最佳实践 +--- + + + + + +### Logging + +有关集群级别日志和应用日志的建议，请参见 [Logging 最佳实践](logging-best-practices.md)。 + +### Monitoring + +配置合理的监控和告警规则对于安全、可靠地运行生产环境中的工作负载至关重要。有关更多建议，请参阅[最佳实践](monitoring-best-practices.md)。 + +### 设置容器的技巧 + +配置良好的容器可以极大地提高环境的整体性能和安全性。有关容器设置的建议，请参见[设置容器的技巧](tips-to-set-up-containers.md)。 + +### Rancher 管理 vSphere 集群的最佳实践 + +[Rancher 管理 vSphere 集群的最佳实践](rancher-managed-clusters-in-vsphere.md)概述了在 vSphere 环境中配置下游 Rancher 集群的参考架构，以及 VMware 记录的标准 vSphere 最佳实践。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-server/rancher-server.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-server/rancher-server.md new file mode 100644 index 00000000000..d297b395a0a --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-server/rancher-server.md @@ -0,0 +1,21 @@ +--- +title: Rancher Server 的最佳实践 +--- + + + + + +本指南介绍了让 Rancher 管理下游 Kubernetes 集群的 Rancher Server 运行建议。 + +### 推荐的架构和基础设施 + +有关在高可用 Kubernetes 集群上设置 Rancher Server 的通用建议，请参见[本指南](tips-for-running-rancher.md)。 + +### 部署策略 + +[本指南](rancher-deployment-strategy.md)旨在帮助你选择部署策略（区域部署/中心辐射型部署），来让 Rancher Server 更好地管理下游 Kubernetes 集群。 + +### 在 vSphere 环境中安装 Rancher + +[本指南](on-premises-rancher-in-vsphere.md)介绍了在 vSphere 环境中安装 Rancher 的参考架构，以及 VMware 记录的标准 vSphere 最佳实践。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md new file mode 100644 index 00000000000..552e79ec76b --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/best-practices/rancher-server/tuning-and-best-practices-for-rancher-at-scale.md @@ -0,0 +1,117 @@ +--- +title: Rancher 大规模部署的调优和最佳实践 +--- + + + + + +本指南介绍了扩容 Rancher 相关场景的最佳实践和调优方法。随着系统规模的不断增长，性能自然会降低，但是可以采取一些步骤将 Rancher 的负载最小化，以优化 Rancher 管理较大基础设施的能力。 + +## 优化 Rancher 性能 + +* 及时升级 Rancher 的 Patch 更新。我们会不断的对 Rancher 进行性能增强和错误修复，最新的 Rancher 版本包含了基于开发人员和用户反馈的所有性能和稳定性的优化改进。 +* 逐渐的执行扩容操作，在操作的过程中观察并监控任何变化，在性能刚出现异常时发现问题并修复，避免其他干扰因素。 +* 尽可能地减少上游 Rancher 集群和下游集群之间的网络延时。需要注意的是，排除掉其他因素，延时与地理距离成正比，如果你的集群或节点分布在全球各地，请考虑改为安装多个 Rancher。 + +## 最小化上游集群的负载 + +在 Rancher 扩容时，一个典型的性能瓶颈是上游（local）Kubernetes 集群的资源增长。上游集群包含所有下游集群的数据信息，许多对下游集群的操作会在上游集群中创建新的对象，并需要在上游集群中进行计算处理。 + +### 最大限度地减少上游集群上的第三方软件 + +大规模运行 Rancher 可能会给内部 Kubernetes 组件（例如 `etcd` 或 `kubeapiserver`）带来大量负载。如果第三方软件干扰了这些组件或 Rancher 的性能，则可能会出现性能问题。 + +每个第三方软件都存在干扰性能的风险。为了防止上游集群出现性能问题，你应该避免运行除 Kubernetes 系统组件和 Rancher 本身之外的任何应用程序或组件。 + +以下类别的软件通常不会干扰 Rancher 或 Kubernetes 系统性能： + * Rancher 内部组件，例如 Fleet + * Rancher 扩展 + * 集群 API 组件 + * CNI 网络插件 + * 云控制器管理器 + * 观察和监控工具（`prometheus-rancher-exporter` 除外） + +除此之外，已发现以下软件会在 Rancher 扩容时影响性能： + * [CrossPlane](https://www.crossplane.io/) + * [Argo CD](https://argoproj.github.io/cd/) + * [Flux](https://fluxcd.io/) + * [prometheus-rancher-exporter](https://github.com/David-VTUK/prometheus-rancher-exporter)（请参阅 [issue 33](https://github.com/David-VTUK/prometheus-rancher-exporter/issues/33)） + +### 管理你的 Object 计数 + +Etcd 是 Kubernetes 和 Rancher 的后端数据库，该数据库可能会遇到单个 Kubernetes 资源类型数量的限制，确切的限制因素各不相同，取决于诸多因素。经验表明，一旦单个资源类型的对象数量超过 60000，通常会出现性能问题，通常该资源类型是 RoleBinding 对象。 + +这在 Rancher 中很常见，因为许多操作会在上游集群中创建新的 RoleBinding 对象。 + +你可以通过以下方法减少上游集群的 `RoleBinding` 数量： +* 减少使用 [Restricted Admin（受限管理员）](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md#受限管理员) 角色，改为使用其他角色。 +* 如果你使用了 [外部认证](../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md)，配置其按组分配角色。 +* 仅在必要情况下将用户添加到集群和项目中。 +* 移除不需要的集群和项目。 +* 仅在必要情况下使用自定义角色。 +* 在自定义角色中，尽可能的减少规则数量。 +* 避免为用户添加多余的角色。 +* 考虑使用数量更少，但性能更强大的集群。 +* Kubernetes 权限始终是 “附加”（允许列表）而不是“减去”（拒绝列表）。尽量减少允许访问集群、项目或命名空间等方面的配置，因为这将导致创建大量的 RoleBinding 对象。 +* 实验一下，创建新项目或集群后，在你的特定用例场景是否表现出更少的 RoleBinding 的效果。 + +### RoleBinding 计数估计 + +预测给定的配置将创建多少个 RoleBinding 对象很复杂，但是以下注意事项可以提供粗略的估算： +* 对于最小的数量估计，请使用公式 `32C + U + 2UaC + 8P + 5Pa`。 + * `C` 是集群数量。 + * `U` 是用户数量。 + * `Ua` 是集群中具有成员资格的用户的平均数量。 + * `P` 是项目总数。 + * `Pa` 是项目中具有成员资格的平均用户数。 +* 受限管理员角色（Restricted Admin）遵循不同的公式，因为具有此角色的每个用户都会额外产生至少 `7C + 2P + 2` 个 `RoleBinding` 对象。 +* * `RoleBinding` 的数量会随着集群、项目和用户的数量线性增加。 + +### 使用新的应用代替旧版应用 + +Rancher 使用两个 Kubernetes 应用程序资源：`apps.projects.cattle.io` 和 `apps.cattle.cattle.io`。以`apps.projects.cattle.io` 为代表的旧版应用程序在低版本的 Cluster Manager UI 中引入，现已弃用。新的应用程序（由 `apps.catalog.cattle.io` 表示）可以在 Cluster Explorer UI 中安装部署。新的 `apps.cattle.cattle.io` 应用程序数据保存在下游集群中，这可以减少上游集群中的资源数量。 + +你应当删除 Cluster Manager UI 中遗留的所有旧版应用程序，并将其替换为 Cluster Explorer UI 中的应用程序。只在 Cluster Explorer UI 中创建任何新应用程序。 + +### 使用授权集群端点 (ACE) + +[授权集群端点](../../../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters.md#4-授权集群端点) (ACE) 提供了 Rancher 部署的 RKE、RKE2 和 K3s 集群的 Kubernetes API 访问。启用后，ACE 会为生成的 `kubeconfig` 文件配置直接访问下游集群 Endpoint，从而绕过 Rancher 代理。在可以直接访问下游集群 Kubernetes API 的场景下，可以减少 Rancher 负载。有关更多信息，请参阅[授权集群端点](../../../reference-guides/rancher-manager-architecture/communicating-with-downstream-user-clusters.md#4-授权集群端点)配置说明。 + +### 减少 Event Handler 执行 + +Rancher 的大部分逻辑发生在 Event Handler 上。每当资源对象产生更新或 Rancher 启动时，这些资源对应的 Event Handler 都会执行。除此之外，它们会每隔 15 小时在 Rancher 计划的同步缓存时再运行一次，这可能会导致 Rancher 运行过程中出现大量性能消耗。可使用 `CATTLE_SYNC_ONLY_CHANGED_OBJECTS` 环境变量禁用计划的 Handler 处理程序执行。如果每 15 小时出现一次性能资源高峰，此设置会有所帮助。 + +`CATTLE_SYNC_ONLY_CHANGED_OBJECTS` 的值可被设置为以下内容，以逗号分隔。这些值代表了处理程序的种类，将处理程序添加到该变量会禁止其在定期的缓存重新同步过程中运行。 + +* `mgmt`：在 Rancher 节点上运行的 Management 管理控制器。 +* `user`：所有集群运行的用户控制器。其中一部分与 Management 管理控制器运行在同一节点上，而另一部分运行在下游集群中，该选项针对的是前者。 +* `scaled`：每个 Rancher 节点上运行的规模控制器。因规模控制器负责关键功能，应当避免设置此值。如果设置可能会破坏集群稳定。 + +简而言之，如果你发现 CPU 使用率每 15 小时出现一次峰值，请将 `CATTLE_SYNC_ONLY_CHANGED_OBJECTS` 环境变量添加到 Rancher Deployment 中（添加至 `spec.containers.env` 列表），其值为 `mgmt,user`。 + +## Rancher 之外的优化 + +集群底层自身的配置也是影响性能的重要因素。如果上游集群存在错误配置，会带来 Rancher 软件所无法解决的性能瓶颈。 + +### 使用 RKE2 直接管理上游集群节点 + +由于 Rancher 对上游集群的要求非常高，尤其是在大规模部署场景，你需要拥有上游集群和节点的所有管理员权限，要找出资源消耗过高的根本原因，请使用标准的 Linux 故障排除工具，这有助于区分是 Rancher、Kubernetes 还是操作系统组件出现的问题。 + +尽管托管 Kubernetes 服务使部署和运行 Kubernetes 集群变得更加容易，但在大规模场景中，不鼓励将其用于上游集群。 托管 Kubernetes 服务通常会限制对单个节点和服务配置的访问。 + +建议在大规模用例场景中使用 RKE2 集群。 + +### 及时更新 Kubernetes 版本 + +你应当及时更新上游集群的 Kubernetes 版本，以确保你的集群具备最新的性能增强和问题修复。 + +### 优化 Etcd + +Etcd 是 Kubernetes 和 Rancher 的后端数据库，在 Rancher 性能中扮演重要的角色。 + +[Etcd 性能](https://etcd.io/docs/v3.4/op-guide/performance/)的两个主要瓶颈是磁盘和网络速度。Etcd 应当在具有高速网络和高读写速度 (IOPS) SSD 硬盘的专用节点上运行。有关 etcd 性能的更多信息，请参阅 [etcd 性能缓慢（性能测试和优化）](https://www.suse.com/support/kb/doc/?id=000020100)和[为大型安装进行 etcd 调优](../../../how-to-guides/advanced-user-guides/tune-etcd-for-large-installs.md)。有关磁盘的信息可以在[安装要求](../../../getting-started/installation-and-upgrade/installation-requirements/installation-requirements.md#磁盘)中找到。 + +根据 etcd 的[复制机制](https://etcd.io/docs/v3.5/faq/#what-is-maximum-cluster-size)，建议在三个节点上运行 etcd，运行在更多的节点上反而会降低速度。 + +Etcd 性能也会受节点之间的网络延迟影响，因此 etcd 节点应与 Rancher 节点部署在一起。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cli-with-rancher/cli-with-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cli-with-rancher/cli-with-rancher.md new file mode 100644 index 00000000000..51f50ec2dca --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cli-with-rancher/cli-with-rancher.md @@ -0,0 +1,9 @@ +--- +title: Rancher CLI +--- + + + + + +Rancher CLI 是一个命令行工具，用于在工作站中与 Rancher 进行交互。以下文档将描述 [Rancher CLI](rancher-cli.md) 和 [kubectl实用程序](kubectl-utility.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/cluster-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/cluster-configuration.md new file mode 100644 index 00000000000..9bc1db8d00b --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/cluster-configuration.md @@ -0,0 +1,32 @@ +--- +title: 集群配置 +--- + + + + + +使用 Rancher 配置 Kubernetes 集群后，你仍然可以编辑集群的选项和设置。 + +有关编辑集群成员资格的信息，请转至[此页面](../../how-to-guides/new-user-guides/manage-clusters/access-clusters/add-users-to-clusters.md)。 + +### 集群配置参考 + +集群配置选项取决于 Kubernetes 集群的类型： + +- [RKE 集群配置](rancher-server-configuration/rke1-cluster-configuration.md) +- [RKE2 集群配置](rancher-server-configuration/rke2-cluster-configuration.md) +- [K3s 集群配置](rancher-server-configuration/k3s-cluster-configuration.md) +- [EKS 集群配置](rancher-server-configuration/eks-cluster-configuration.md) +- [GKE 集群配置](gke-cluster-configuration.md) +- [AKS 集群配置](rancher-server-configuration/aks-cluster-configuration.md) + +### 不同类型集群的管理功能 + +对于已有集群而言，可提供的选项和设置取决于你配置集群的方法。 + +下表总结了每一种类型的集群和对应的可编辑的选项和设置： + +import ClusterCapabilitiesTable from '../../shared-files/_cluster-capabilities-table.md'; + + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/downstream-cluster-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/downstream-cluster-configuration.md new file mode 100644 index 00000000000..591285c378e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/downstream-cluster-configuration.md @@ -0,0 +1,9 @@ +--- +title: 下游集群配置 +--- + + + + + +以下文档将讨论[节点模板配置](./node-template-configuration.md)和[主机配置](./machine-configuration.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/machine-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/machine-configuration.md new file mode 100644 index 00000000000..7669ec23377 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/machine-configuration/machine-configuration.md @@ -0,0 +1,9 @@ +--- +title: 主机配置 +--- + + + + + +主机配置指的是如何将资源分配给虚拟机。请参阅 [Amazon EC2](amazon-ec2.md)、[DigitalOcean](digitalocean.md) 和 [Azure](azure.md) 的文档以了解更多信息。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/node-template-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/node-template-configuration.md new file mode 100644 index 00000000000..35779a5d3ae --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/node-template-configuration.md @@ -0,0 +1,9 @@ +--- +title: 节点模板配置 +--- + + + + + +要了解节点模板配置，请参阅[EC2 节点模板配置](amazon-ec2.md)、[DigitalOcean 节点模板配置](digitalocean.md)、[Azure 节点模板配置](azure.md)、[vSphere 节点模板配置](vsphere.md)和 [Nutanix 节点模板配置](nutanix.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md index 7ac1c106d1d..a8956d548d5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md @@ -13,7 +13,7 @@ title: vSphere 节点模板配置 | 凭证字段 | 描述 | |-----------------|--------------| | vCenter 或 ESXi Server | 输入 vCenter 或 ESXi 主机名/IP。ESXi 是你创建和运行虚拟机和虚拟设备的虚拟化平台。你可以通过 vCenter Server 服务来管理网络中连接的多个主机并池化主机资源。 | -| 端口 | 配置 vCenter 或 ESXi Server 的端口。 | +| 端口 | 可选：配置 vCenter 或 ESXi Server 的端口。 | | 用户名和密码 | 你的 vSphere 登录用户名和密码。 | ## 调度 @@ -92,4 +92,4 @@ title: vSphere 节点模板配置 如果要配置 Red Hat Enterprise Linux (RHEL) 或 CentOS 节点，请将 **Docker Install URL** 字段保留为默认值，或选择 **none**。由于 Docker 已经安装在这些节点上，因此将绕过 Docker 安装检查。 如果没有将 **Docker Install URL** 设置为默认值或 **none**，你可能会看到错误消息：`Error creating machine: RHEL ssh command error: command: sudo -E yum install -y curl err: exit status 1 output: Updating Subscription Management repositories`。 -::: +::: \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md new file mode 100644 index 00000000000..80081aefe75 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-cluster-configuration.md @@ -0,0 +1,324 @@ +--- +title: GKE 集群配置参考 +--- + + + + + +## Rancher 2.6 变更 + +- 支持额外的配置选项： + - 项目网络隔离 + - 网络标签 + +## 集群位置 + +| 值 | 描述 | +|--------|--------------| +| 位置类型 | 地区 (zone) 或区域 (region)。借助 GKE，你可以根据工作负载的可用性要求和预算创建一个量身定制的集群。默认情况下，集群的节点在单个计算区域中运行。选择多个区域时，集群的节点将跨越多个计算区域，而 controlplane 则只位于单个区域中。区域集群也增加了 controlplane 的可用性。有关选择集群可用性类型的帮助，请参阅[这些文档](https://cloud.google.com/kubernetes-engine/docs/best-practices/scalability#choosing_a_regional_or_zonal_control_plane)。 | +| 地区 | 计算引擎中的每个区域都包含多地区。有关可用区域和可用区的更多信息，请参阅[这些文档](https://cloud.google.com/compute/docs/regions-zones#available)。 | +| 其他地区 | 对于地区性集群，你可以选择其他地区来创建[多地区集群](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#multi-zonal_clusters)。 | +| 区域 | 对于[区域性集群](https://cloud.google.com/kubernetes-engine/docs/concepts/types-of-clusters#regional_clusters)，你可以选择一个区域。有关可用区域和可用区的更多信息，请参阅[本节](https://cloud.google.com/compute/docs/regions-zones#available)。地区名称的前面部分是区域的名称。 | + +## 集群选项 + +### Kubernetes 版本 + +_可变：是_ + +有关 GKE Kubernetes 版本的更多信息，请参阅[这些文档](https://cloud.google.com/kubernetes-engine/versioning)。 + +### 容器地址范围 + +_可变：否_ + +集群中 Pod 的 IP 地址范围。必须是有效的 CIDR 范围，例如 10.42.0.0/16。如果未指定，则会自动从 10.0.0.0/8 中选择一个随机范围，并排除已分配给 VM、其他集群或路由的范围。自动选择的范围可能与预留的 IP 地址、动态路由或与集群对等的 VPC 中的路由发生冲突。 + +### 网络 + +_可变：否_ + +集群连接的 Compute Engine 网络。将使用此网络创建路由和防火墙。如果使用[共享 VPC](https://cloud.google.com/vpc/docs/shared-vpc)，与你的项目共享的 VPC 网络将显示在此处。你将可以在此字段中进行选择。有关详细信息，请参阅[此页面](https://cloud.google.com/vpc/docs/vpc#vpc_networks_and_subnets)。 + +### 节点子网/子网 + +_可变：否_ + +集群连接到的 Compute Engine 子网。该子网必须属于**网络**字段中指定的网络。选择一个现有的子网，或选择“自动创建子网”来自动创建一个子网。如果不使用现有网络，则需要使用**子网名称**来生成一个。如果使用[共享 VPC](https://cloud.google.com/vpc/docs/shared-vpc)，与你的项目共享的 VPC 子网将显示在此处。如果使用共享 VPC 网络，则无法选择“自动创建子网”。如需更多信息，请参阅[此页面](https://cloud.google.com/vpc/docs/vpc#vpc_networks_and_subnets)。 + +### 子网名称 + +_可变：否_ + +使用提供的名称自动创建子网。如果为**节点子网**或**子网**选择了“自动创建子网”，则为必填。有关子网的更多信息，请参阅[此页面](https://cloud.google.com/vpc/docs/vpc#vpc_networks_and_subnets)。 + +### IP 别名 + +_可变：否_ + +启用[别名 IP](https://cloud.google.com/vpc/docs/alias-ip)。这将启用 VPC 原生流量路由。如果使用[共享 VPC](https://cloud.google.com/vpc/docs/shared-vpc)，则为必填。 + +### 网络策略 + +_可变：是_ + +在集群上启用的网络策略。网络策略定义了集群中 pod 和 service 之间可以发生的通信级别。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/how-to/network-policy)。 + +### 项目网络隔离 + +_可变：是_ + +选择启用或禁用项目间通信。请注意，如果启用**项目网络隔离**，则将自动启用**网络策略**和**网络策略配置**，反之则不然。 + +### 节点 IPv4 CIDR 块 + +_可变：否_ + +此集群中实例 IP 的 IP 地址范围。如果为**节点子网**或**子网**选择了“自动创建子网”，则可以进行设置。必须是有效的 CIDR 范围，例如 10.96.0.0/14。有关如何确定 IP 地址范围的详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing)。 + +### 集群次要范围名称 + +_可变：否_ + +Pod IP 地址的现有次要范围的名称。如果选中，将自动填充**集群 Pod 地址范围**。如果使用共享 VPC 网络，则为必填。 + +### 集群 Pod 地址范围 + +_可变：否_ + +分配给集群中 pod 的 IP 地址范围。必须是有效的 CIDR 范围，例如 10.96.0.0/11。如果未提供，将自动创建。如果使用共享 VPC 网络，则必须提供。有关如何确定 pod 的 IP 地址范围的更多信息，请参阅[本节](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing_secondary_range_pods)。 + +### Service 次要范围名称 + +_可变：否_ + +Service IP 地址的现有次要范围的名称。如果选中，将自动填充 **Service 地址范围**。如果使用共享 VPC 网络，则为必填。 + +### Service 地址范围 + +_可变：否_ + +分配给集群中 Service 的地址范围。必须是有效的 CIDR 范围，例如 10.94.0.0/18。如果未提供，将自动创建。如果使用共享 VPC 网络，则必须提供。有关如何确定 Service 的 IP 地址范围的详细信息，请参阅[本节](https://cloud.google.com/kubernetes-engine/docs/concepts/alias-ips#cluster_sizing_secondary_range_svcs)。 + +### 私有集群 + +_可变：否_ + +:::caution + +私有集群需要在 Rancher 之外进行额外的规划和配置。请参阅[私有集群指南](gke-private-clusters.md)。 + +::: + +仅分配节点内部 IP 地址。除非在 GCP 中执行了额外的联网步骤，否则私有集群节点无法访问公共互联网。 + +### 启用私有端点 + +:::caution + +私有集群需要在 Rancher 之外进行额外的规划和配置。请参阅[私有集群指南](gke-private-clusters.md)。 + +::: + +_可变：否_ + +锁定对 controlplane 端点的外部访问。仅当**私有集群**也被选中时可用。如果选中，并且 Rancher 无法直接访问集群所在的虚拟私有云网络，Rancher 将提供在集群上运行的注册命令，以使 Rancher 能够连接到集群。 + +### 主 IPV4 CIDR 块 + +_可变：否_ + +controlplane VPC 的 IP 范围。 + +### 主授权网络 + +_可变：是_ + +启用 controlplane 授权网络，以阻止不受信任的非 GCP 源 IP 通过 HTTPS 访问 Kubernetes master。如果选择，则可以添加额外的授权网络。如果集群是使用公共端点创建的，则此选项可用于将公共端点的访问锁定到特定网络（例如运行 Rancher 服务的网络）。如果集群只有一个私有端点，则需要此设置。 + +## 其他选项 + +### 集群插件 + +其他 Kubernetes 集群组件。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/reference/rest/v1/projects.locations.clusters#Cluster.AddonsConfig)。 + +#### 水平 Pod 自动缩放 + +_可变：是_ + +Horizo​​ntal Pod Autoscaler 通过自动增加或减少 Pod 的数量来调整 Kubernetes 工作负载，从而响应工作负载的 CPU 或内存消耗，以及 Kubernetes 内部报告的自定义指标或集群外部设置的指标。详情请参见[本页面](https://cloud.google.com/kubernetes-engine/docs/concepts/horizontalpodautoscaler)。 + +#### HTTP (L7) 负载均衡 + +_可变：是_ + +HTTP (L7) 负载均衡将 HTTP 和 HTTPS 流量分配到托管在 GKE 上的后端。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/tutorials/http-balancer)。 + +#### 网络策略配置（仅限 master） + +_可变：是_ + +NetworkPolicy 的配置。仅跟踪 master 节点上是否启用了插件，不跟踪是否为节点启用了网络策略。 + +### 集群特征（Alpha 功能） + +_可变：否_ + +打开集群的所有 Kubernetes alpha API 组和功能。启用后，集群无法升级，并且会在 30 天后自动删除。由于 GKE SLA 未支持 alpha 集群，因此不建议将 Alpha 集群用于生产环境。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/concepts/alpha-clusters)。 + +### Logging 服务 + +_可变：是_ + +集群用于写入日志的日志管理服务。要么使用 [Cloud Logging](https://cloud.google.com/logging)，要么不使用日志管理服务（不会从集群中导出日志）。 + +### 监控服务 + +_可变：是_ + +集群用于写入指标的监控服务。要么使用 [Cloud Monitoring](https://cloud.google.com/monitoring)，要么不使用集群监控服务（不会从集群中导出指标）。 + + +### 维护窗口 + +_可变：是_ + +设置时长 4 小时的维护窗口的开始时间。使用 HH:MM 格式在 UTC 时区中指定时间。有关详细信息，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/concepts/maintenance-windows-and-exclusions)。 + +## 节点池 + +在此部分中，输入描述节点池中每个节点的配置的详细信息。 + +### Kubernetes 版本 + +_可变：是_ + +节点池中每个节点的 Kubernetes 版本。有关 GKE Kubernetes 版本的更多信息，请参阅[这些文档](https://cloud.google.com/kubernetes-engine/versioning)。 + +### 镜像类型 + +_可变：是_ + +节点操作系统镜像。有关 GKE 为每个操作系统提供的节点镜像选项，请参阅[此页面](https://cloud.google.com/kubernetes-engine/docs/concepts/node-images#available_node_images)。 + +:::note + +默认选项是 “Container-Optimized OS with Docker”。GCP Container-Optimized OS 上的只读文件系统与 Rancher 中的 [legacy logging](/versioned_docs/version-2.0-2.4/explanations/integrations-in-rancher/cluster-logging/cluster-logging.md) 实现不兼容。如果你需要使用旧版日志管理功能，请选择 “Ubuntu with Docker” 或 “Ubuntu with Containerd”。[current logging feature](../../../../integrations-in-rancher/logging/logging.md) 与 Container-Optimized OS 镜像兼容。 + +::: + +:::note + +如果节点池镜像类型选择 “Windows Long Term Service Channel” 或 “Windows Semi-Annual Channel”，还必须至少添加一个 Container-Optimized OS 或 Ubuntu 节点池。 + +::: + +### 主机类型 + +_可变：否_ + +节点实例可用的虚拟化硬件资源。有关 Google Cloud 主机类型的详细信息，请参阅[此页面](https://cloud.google.com/compute/docs/machine-types#machine_types)。 + +### 根磁盘类型 + +_可变：否_ + +标准永久性磁盘由标准磁盘驱动器 (HDD) 支持，而 SSD 永久性磁盘由固态硬盘 (SSD) 支持。有关详细信息，请参阅[本节](https://cloud.google.com/compute/docs/disks)。 + +### 本地 SSD 磁盘 + +_可变：否_ + +配置每个节点的本地 SSD 磁盘存储（以 GB 为单位）。本地 SSD 物理连接到托管你的 VM 实例的服务器。与标准永久性磁盘或 SSD 永久性磁盘相比，本地 SSD 具有更高的吞吐量和更低的延迟。存储在本地 SSD 上的数据只会保留到实例停止或删除。有关详细信息，请参阅[本节](https://cloud.google.com/compute/docs/disks#localssds)。 + +### 抢占式节点（beta） + +_可变：否_ + +抢占式节点也称为抢占式虚拟机。通常是最长持续 24 小时的 Compute Engine 虚拟机实例，不提供可用性保证。详情请参见[本页面](https://cloud.google.com/kubernetes-engine/docs/how-to/preemptible-vms)。 + +### 污点 + +_可变：否_ + +将污点应用于节点时，仅允许容忍该污点的 Pod 在该节点上运行。在 GKE 集群中，你可以将污点应用到节点池，这会将污点应用到池中的所有节点。 + +### 节点标签 + +_可变：否_ + +你可以将标签应用到节点池，这会将标签应用到池中的所有节点。 + +无效标签会阻止升级，或阻止 Rancher 启动。有关标签语法的详细信息，请参阅 [Kubernetes 文档](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set)。 + +### 网络标签 + +_可变：否_ + +你可以将网络标签添加到节点池以制定防火墙规则和子网之间的路由。标签将应用于池中的所有节点。 + +有关标签语法和要求的详细信息，请参阅 [Kubernetes 文档](https://cloud.google.com/vpc/docs/add-remove-network-tags)。 + +## 组详细信息 + +在此部分中，输入描述节点池的详细信息。 + +### 名称 + +_可变：否_ + +输入节点池的名称。 + +### 初始节点数 + +_可变：是_ + +节点池中初始节点数的整数。 + +### 每个节点的最大 Pod 数量 + +_可变：否_ + +GKE 的硬性限制是每个节点 110 个 Pod。有关 Kubernetes 限制的更多信息，请参阅[本节](https://cloud.google.com/kubernetes-engine/docs/best-practices/scalability#dimension_limits)。 + +### 自动缩放 + +_可变：是_ + +节点池自动缩放会根据工作负载的需求动态创建或删除节点。详情请参见[本页面](https://cloud.google.com/kubernetes-engine/docs/concepts/cluster-autoscaler)。 + +### 自动修复 + +_可变：是_ + +GKE 的节点自动修复功能可帮助你将集群中的节点保持在健康的运行状态。启用后，GKE 会定期检查集群中每个节点的运行状况。如果某个节点在较长时间段内连续未通过健康检查，GKE 会为该节点启动修复过程。有关详细信息，请参阅[自动修复节点](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-repair)。 + +### 自动升级 + +_可变：是_ + +启用后，当你的 controlplane [按照你的需求更新](https://cloud.google.com/kubernetes-engine/upgrades#automatic_cp_upgrades)时，自动升级功能会使集群中的节点与集群 controlplane（master）版本保持同步。有关自动升级节点的更多信息，参见[此页面。](https://cloud.google.com/kubernetes-engine/docs/how-to/node-auto-upgrades) + +### 访问范围 + +_可变：否_ + +设置访问范围是为你的节点指定权限的旧版方法。 + +- **允许默认访问**：新集群的默认访问是 [Compute Engine 默认 ServiceAccount](https://cloud.google.com/compute/docs/access/service-accounts?hl=en_US#default_service_account)。 +- **允许完全访问所有 Cloud API**：通常，你只需设置云平台访问范围来允许完全访问所有 Cloud API，然后仅授予 ServiceAccount 相关的 IAM 角色。授予虚拟机实例的访问范围和授予 ServiceAccount 的 IAM 角色的组合决定了 ServiceAccount 对该实例的访问量。 +- **为每个 API 设置访问权限**：或者，你可以设置服务将调用的特定 API 方法的访问范围。 + +有关详细信息，请参阅[为 VM 启用 ServiceAccount](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances)。 + + +### 配置刷新间隔 + +刷新间隔可以通过 “gke-refresh” 来配置，它是一个代表秒的整数。 + +默认值为 300 秒。 + +你可以通过运行 `kubectl edit setting gke-refresh` 来更改同步间隔。 + +刷新窗口越短，争用条件发生的可能性就越小。但这确实增加了遇到 GCP API 可能存在的请求限制的可能性。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-private-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-private-clusters.md index ff4638fc589..fa562ae9cee 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-private-clusters.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/gke-cluster-configuration/gke-private-clusters.md @@ -26,7 +26,7 @@ Cloud NAT 将[产生费用](https://cloud.google.com/nat/pricing)。 ::: -如果要求限制节点的传入和传出流量，请按照离线安装说明，在集群所在的 VPC 上设置一个私有容器[镜像仓库](https://rancher.com/docs/rancher/v2.6/en/installation/other-installation-methods/air-gap/)，从而允许集群节点访问和下载运行 cluster agent 所需的镜像。如果 controlplane 端点也是私有的，Rancher 将需要[直接访问](#直接访问)它。 +如果要求限制节点的传入和传出流量，请按照离线安装说明，在集群所在的 VPC 上设置一个私有容器[镜像仓库](../../../../pages-for-subheaders/air-gapped-helm-cli-install.md)，从而允许集群节点访问和下载运行 cluster agent 所需的镜像。如果 controlplane 端点也是私有的，Rancher 将需要[直接访问](#直接访问)它。 ### 私有 controlplane 端点 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/rancher-server-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/rancher-server-configuration.md new file mode 100644 index 00000000000..bde06845d60 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/rancher-server-configuration.md @@ -0,0 +1,16 @@ +--- +title: Rancher Server Configuration +--- + + + + + +- [RKE1 集群配置](rke1-cluster-configuration.md) +- [RKE2 集群配置](rke2-cluster-configuration.md) +- [K3s 集群配置](k3s-cluster-configuration.md) +- [EKS 集群配置](eks-cluster-configuration.md) +- [AKS 集群配置](aks-cluster-configuration.md) +- [GKE 集群配置](gke-cluster-configuration/gke-cluster-configuration.md) +- [使用现有节点](use-existing-nodes/use-existing-nodes.md) +- [同步集群](sync-clusters.md) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/rke2-cluster-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/rke2-cluster-configuration.md index c1771ed32d3..3f4328512ae 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/rke2-cluster-configuration.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/rke2-cluster-configuration.md @@ -160,7 +160,7 @@ Rancher 与以下开箱即用的网络提供商兼容： ### Agent 环境变量 -为 [Rancher agent](https://rancher.com/docs/rancher/v2.6/en/cluster-provisioning/rke-clusters/rancher-agents/) 设置环境变量的选项。你可以使用键值对设置环境变量。有关详细信息，请参阅 [RKE2 文档](https://docs.rke2.io/reference/linux_agent_config)。 +为 [Rancher agent](../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md) 设置环境变量的选项。你可以使用键值对设置环境变量。有关详细信息，请参阅 [RKE2 文档](https://docs.rke2.io/reference/linux_agent_config)。 ### etcd diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md new file mode 100644 index 00000000000..c4e1efd5335 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md @@ -0,0 +1,141 @@ +--- +title: 在现有自定义节点上启动 Kubernetes +description: 要创建具有自定义节点的集群，你需要访问集群中的服务器，并根据 Rancher 的要求配置服务器。 +--- + + + + + +创建自定义集群时，Rancher 使用 RKE（Rancher Kubernetes Engine）在本地裸机服务器、本地虚拟机或云服务器节点中创建 Kubernetes 集群。 + +要使用此选项，你需要访问要在 Kubernetes 集群中使用的服务器。请根据[要求](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)配置每台服务器，其中包括硬件要求和 Docker 要求。在每台服务器上安装 Docker 后，你还需要在每台服务器上运行 Rancher UI 中提供的命令，从而将每台服务器转换为 Kubernetes 节点。 + +本节介绍如何设置自定义集群。 + +## 使用自定义节点创建集群 + +:::note 使用 Windows 主机作为 Kubernetes Worker 节点？ + +在开始之前，请参阅[配置 Windows 自定义集群](use-windows-clusters.md)。 + +::: + +### 1. 配置 Linux 主机 + +你可以通过配置 Linux 主机，来创建自定义集群。你的主机可以是： + +- 云虚拟机 +- 本地虚拟机 +- 裸机服务器 + +如果要重复使用之前的自定义集群中的节点，请在复用之前[清理节点](../../../../how-to-guides/new-user-guides/manage-clusters/clean-cluster-nodes.md)。如果你重复使用尚未清理的节点，则集群配置可能会失败。 + +根据[安装要求](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)和[生产就绪集群的检查清单](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/checklist-for-production-ready-clusters.md)配置主机。 + +如果你使用 Amazon EC2 作为主机，并希望使用[双栈 (dual-stack)](https://kubernetes.io/docs/concepts/services-networking/dual-stack/) 功能，则需要满足配置主机的其他[要求](https://rancher.com/docs/rke//latest/en/config-options/dual-stack#requirements)。 + +### 2. 创建自定义集群 + +1. 点击 **☰ > 集群管理**。 +1. 在**集群**页面上，单击**创建**。 +1. 单击**自定义**。 +1. 输入**集群名称**。 +1. 在**集群配置**中，选择 Kubernetes 版本、要使用的网络提供商，以及是否启用项目网络隔离。要查看更多集群选项，请单击**显示高级选项**。 + + :::note 你使用 Windows 主机作为 Kubernetes Worker 节点？ + + - 请参阅[启用 Windows 支持选项](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)。 + - 支持 Windows 集群的唯一网络插件是 Flannel。 + + ::: + + :::note Amazon EC2 上的双栈： + + 如果你使用 Amazon EC2 作为主机，并希望使用[双栈 (dual-stack)](https://kubernetes.io/docs/concepts/services-networking/dual-stack/) 功能，则需要满足配置 RKE 的其他[要求](https://rancher.com/docs/rke//latest/en/config-options/dual-stack#requirements)。 + + ::: + +6. 点击**下一步**。 + +4. 使用**成员角色**为集群配置用户授权。点击**添加成员**添加可以访问集群的用户。使用**角色**下拉菜单为每个用户设置权限。 + +7. 从**节点角色**中，选择要由集群节点充当的角色。你必须为 `etcd`、`worker` 和 `controlplane` 角色配置至少一个节点。自定义集群需要所有三个角色才能完成配置。有关角色的详细信息，请参阅[本节](../../../kubernetes-concepts.md#kubernetes-集群中节点的角色)。 + +:::note + +- 使用 Windows 主机作为 Kubernetes Worker 节点？请参阅[本节](../../../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)。 +- 裸机服务器提醒：如果你想将裸机服务器专用于每个角色，则必须为每个角色配置一个裸机服务器（即配置多个裸机服务器）。 + +::: + +8. **可选**：点击[显示高级选项](../../../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md)来指定注册节点时使用的 IP 地址，覆盖节点的主机名，或将[标签](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)或[污点](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)添加到节点。 + +9. 将屏幕上显示的命令复制到剪贴板。 + +10. 使用你惯用的 shell（例如 PuTTy 或远程终端）登录到你的 Linux 主机。粘贴剪贴板的命令并运行。 + +:::note + +如果要将特定主机专用于特定节点角色，请重复步骤 7-10。根据需要多次重复这些步骤。 + +::: + +11. 在 Linux 主机上运行完命令后，单击**完成**。 + +**结果**： + +你已创建集群，集群的状态是**配置中**。Rancher 已在你的集群中。 + +当集群状态变为 **Active** 后，你可访问集群。 + +**Active** 状态的集群会分配到两个项目： + +- `Default`：包含 `default` 命名空间 +- `System`：包含 `cattle-system`，`ingress-nginx`，`kube-public` 和 `kube-system` 命名空间。 + + +### 3. 仅限亚马逊：标签资源 + +如果你已将集群配置为使用 Amazon 作为**云提供商**，请使用集群 ID 标记你的 AWS 资源。 + +[Amazon 文档：标记你的 Amazon EC2 资源](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html) + +:::note + +你可以使用 Amazon EC2 实例，而无需在 Kubernetes 中配置云提供商。如果你想使用特定的 Kubernetes 云提供商功能，配置云提供商即可。如需更多信息，请参阅 [Kubernetes 云提供商](https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/)。 + +::: + +以下资源需要使用 `ClusterID` 进行标记： + +- **Nodes**：Rancher 中添加的所有主机。 +- **Subnet**：集群使用的子网。 +- **Security Group**：用于你的集群的安全组。 + +:::note + +不要标记多个安全组。创建 Elastic Load Balancer 时，标记多个组会导致错误。 + +::: + +应该使用的标签是： + +``` +Key=kubernetes.io/cluster/, Value=owned +``` + +`` 可以是你选择的任何字符串。但是，必须在你标记的每个资源上使用相同的字符串。将值设置为 `owned` 会通知集群所有带有 `` 标记的资源都由该集群拥有和管理。 + +如果你在集群之间共享资源，你可以将标签更改为： + +``` +Key=kubernetes.io/cluster/CLUSTERID, Value=shared +``` + +## 可选的后续步骤 + +创建集群后，你可以通过 Rancher UI 访问集群。最佳实践建议你设置以下访问集群的备用方式： + +- **通过 kubectl CLI 访问你的集群**：按照[这些步骤](../../../../how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#accessing-clusters-with-kubectl-from-your-workstation)在你的工作站上使用 kubectl 访问集群。在这种情况下，你将通过 Rancher Server 的认证代理进行认证，然后 Rancher 会让你连接到下游集群。此方法允许你在没有 Rancher UI 的情况下管理集群。 +- **通过 kubectl CLI 使用授权的集群端点访问你的集群**：按照[这些步骤](../../../../how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md#authenticating-directly-with-a-downstream-cluster)直接使用 kubectl 访问集群，而无需通过 Rancher 进行认证。我们建议设置此替代方法来访问集群，以便在无法连接到 Rancher 时访问集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md new file mode 100644 index 00000000000..1110b88e03a --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/monitoring-v2-configuration/monitoring-v2-configuration.md @@ -0,0 +1,15 @@ +--- +title: Monitoring V2 配置 +--- + + + + + +以下将说明在 Rancher 中如何配置 Monitoring V2 的基本重要选项： + +- [接收器配置](receivers.md) +- [路由配置](routes.md) +- [ServiceMonitor 和 PodMonitor 配置](servicemonitors-and-podmonitors.md) +- [Helm Chart 选项](helm-chart-options.md) +- [示例](examples.md) \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/monitoring-v2-configuration/routes.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/monitoring-v2-configuration/routes.md index 2a3da893016..0c39e576670 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/monitoring-v2-configuration/routes.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/monitoring-v2-configuration/routes.md @@ -42,7 +42,7 @@ Alerting Drivers 为 Alertmanager 将告警代理到非原生接收器，例如 | 字段 | 默认 | 描述 | |-------|--------------|---------| -| 分组依据 | N/A | 用于分组的标签列表。标签必须是唯一的。如果提供了特殊标签“...”（由所有可能的标签聚合），标签必须在列表中是唯一的元素。接受字符串列表。有关详细信息，请参阅[上游文档](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#route)。 | +| 分组依据 | N/A | 用于分组的标签列表。所有标签都必须是唯一的。如果提供了特殊标签“...”（由所有可能的标签聚合），标签必须在列表中是唯一的元素。接受字符串列表。有关详细信息，请参阅[上游文档](https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#route)。 | | 组等待时长 | 30s | 在发送之前，缓冲同一组告警的等待时间。 | | 组间隔 | 5m | 等待多长时间才发送已添加到告警组的告警，其中该告警组的初次通知已被发送。 | | 重复间隔 | 4h | 等待多长时间后，才重新发送已发送的告警。 | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/prometheus-federator/prometheus-federator.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/prometheus-federator/prometheus-federator.md new file mode 100644 index 00000000000..6e4c4bd8ac3 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/prometheus-federator/prometheus-federator.md @@ -0,0 +1,109 @@ +--- +title: Prometheus Federator +--- + + + + + +Prometheus Federator（也称为 Project Monitoring V2）基于 [rancher/helm-project-operator](https://github.com/rancher/helm-project-operator) 部署一个 Helm Project Operator。该 Operator 管理 Helm Chart 的部署，每个 Operator 都包含一个 Project Monitoring Stack，而每个堆栈都包含： + +- [Prometheus](https://prometheus.io/)（由 [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) 在外部管理） +- [Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/)（由 [Prometheus Operator](https://github.com/prometheus-operator/prometheus-operator) 在外部管理） +- [Grafana](https://github.com/helm/charts/tree/master/stable/grafana)（通过嵌入式 Helm Chart 部署） +- 基于 [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus/) 社区策划资源集合的默认 PrometheusRules 和 Grafana 仪表板 +- 监视已部署资源的默认 ServiceMonitor + +:::note 重要提示： + +Prometheus Federator 适合在已安装 Prometheus Operator CRD 的集群中与现有的 Prometheus Operator Deployment 一起部署。 + +::: + +## Operator 工作原理 + +1. 在部署此 Chart 时，用户可以创建 ProjectHelmCharts CR，并在**项目 Registration 命名空间 (`cattle-project-`)** 中将 `spec.helmApiVersion` 设置为 `monitoring.cattle.io/v1alpha1`（在 Rancher UI 中也称为“项目监控”）。 +2. 在看到每个 ProjectHelmChartCR 时，Operator 会代表项目所有者在**项目 Release 命名空间 (`cattle-project--monitoring`)** 中自动部署一个 Project Prometheus 堆栈（基于 ProjectHelmChart 控制器在 **Operator / System Namespace** 中创建的 HelmChart CR 和 HelmRelease CR）。 +3. RBAC 将自动分配到项目 Release 命名空间中，从而允许用户查看 Prometheus、Alertmanager 以及已部署的 Project Monitoring Stack 的 Grafana UI（基于在项目 Registration 命名空间上针对[面向用户的默认 Kubernetes 角色](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles)定义的 RBAC）。有关详细信息，请参阅[配置 RBAC](rbac.md)。 + +### 什么是项目？ + +在 Prometheus Federator 中，项目是一组可以由 `metav1.LabelSelector` 标识的命名空间。默认情况下，用于标识项目的标签是 `field.cattle.io/projectId`，该标签用于标识给定 Rancher 项目中包含的命名空间。 + +### 配置由 ProjectHelmChart 创建的 Helm 版本 + +此 ProjectHelmChart 资源的 `spec.values` 对应于为底层 Helm Chart 配置的 `values.yaml` 覆盖，该 Helm Chart 是 Operator 代表用户部署的。要查看底层 Chart 的 `values.yaml` 规范，你可以选择以下其中一种方式： + +- 查看位于 [`charts/rancher-project-monitoring` 中的 `rancher/prometheus-federator`](https://github.com/rancher/prometheus-federator/blob/main/charts/rancher-project-monitoring) 的 Chart 定义（Chart 版本会绑定到 Operator 版本）。 +- 查找在每个项目 Registration 命名空间中自动创建的名为 `monitoring.cattle.io.v1alpha1` 的 ConfigMap，其中包含用于配置 Chart（直接嵌入到 `prometheus-federator` 二进制文件中）的`values.yaml` 和 `questions.yaml`。 + +### 命名空间 + +Prometheus Federator 是基于 [rancher/helm-project-operator](https://github.com/rancher/helm-project-operator) 的 Project Operator，Prometheus Federator 提供了三类命名空间供 Operator 查找： + +1. **Operator / System 命名空间**：部署 Operator 的命名空间（例如 `cattle-monitoring-system`）。此命名空间将包含该 Operator 监视的所有 ProjectHelmChart 的所有 HelmChart 和 HelmRelease。**只有集群管理员才能访问此命名空间**。 + +2. **项目 Registration 命名空间 (`cattle-project-`)**：Operator 在这些命名空间中监视 ProjectHelmChart。对于在项目发布命名空​​间中创建的自动分配的 RBAC，应用于此命名空间的 RoleBinding 和 ClusterRoleBinding 也会作为 RBAC 的真实来源。有关详细信息，请参阅 [RBAC 页面](rbac.md)。**项目所有者（admin）、项目成员（edit）和只读成员（view）应该有权访问此命名空间。** + + :::note 注意事项： + + - 如果提供了 `.Values.global.cattle.projectLabel`（默认设置为 `field.cattle.io/projectId`），则 Operator 会自动生成项目注册命名空间，并将它导入到命名空间绑定的项目中。换言之，如果观察到至少一个带有该标签的命名空间，则 Operator 会创建一个项目 Registration 命名空间。除非出现以下两种情况，否则 Operator 不会让这些命名空间被删除。第一种情况是带有该标签的所有命名空间都消失了（例如，这是该项目中的最后一个命名空间，在这种情况下，命名空间将标有标签 `"helm.cattle.io/helm-project-operator-orphaned": "true"`，表示可以删除）。第二种情况是由于项目 ID 是在 `.Values.helmProjectOperator.otherSystemProjectLabelValues` 下提供的（用作项目的拒绝名单），导致 Operator 不再监视该项目。这些命名空间不会被自动删除，这样能避免破坏用户数据。如果需要，建议用户在创建或删除项目时手动清理这些命名空间。 + + - 如果未提供 `.Values.global.cattle.projectLabel`，则 Operator / System 命名空间也是项目注册命名空间。 + + ::: + +3. **项目发布命名空​​间（`cattle-project--monitoring`）**：Operator 代表 ProjectHelmChart 在其中部署项目监控堆栈的命名空间集。Operator 还将根据在项目 Registration 命名空间中找到的绑定，自动为项目监控堆栈在此命名空间中创建的角色分配 RBAC。**只有集群管理员才能访问这个命名空间。部署的 Helm Chart 和 Prometheus Federator 将为项目所有者（admin）、项目成员（edit）和只读成员（view）分配该命名空间的有限访问权限。** + + :::note 注意事项： + + - 项目发布命名空间会自动部署并导入到 ID 在 `.Values.helmProjectOperator.projectReleaseNamespaces.labelValue` 下指定的项目中，如果未指定，且项目注册命名空间中指定了 ProjectHelmChart，则默认为 `.Values.global.cattle.systemProjectId` 的值。 + + - 项目发布命名空​​间的孤立约定与项目注册命名空间的相同（参见上面的注释）。 + + - 如果 `.Values.projectReleaseNamespaces.enabled` 为 false，则项目发布命名空​​间与项目注册命名空间是相同的。 + + ::: + +### Helm 资源（HelmChart、HelmRelease） + +在部署 ProjectHelmChart 时，Prometheus Federator 将自动创建和管理两个子自定义资源，它们依次管理以下底层 Helm 资源： + +- HelmChart CR（通过 Operator 中的嵌入式 [k3s-io​​/helm-contoller](https://github.com/k3s-io/helm-controller) 管理）：此自定义资源会根据应用到 HelmChart CR 的变更，在触发 `helm install`、`helm upgrade` 或 `helm uninstall` 的同一命名空间中自动创建一个 Job。此 CR 会根据 ProjectHelmChart 的更改（例如，修改 `values.yaml`）或底层项目定义的更改（例如，从项目中添加或删除命名空间）自动更新。 + +:::note 重要提示： + +如果 ProjectHelmChart 没有部署或更新底层项目监控堆栈，你可以先使用此资源在 Operator / System 命名空间中创建的 Job 来检查 Helm 操作是否有问题。通常只能由**集群管理员访问**。 + +::: + +- HelmRelease CR（通过 Operator 中的嵌入式 [rancher/helm-locker](https://github.com/rancher/helm-locker) 管理）：此自定义资源会自动锁定已部署的 Helm 版本并自动覆盖对底层资源的更新，除非更改是 Helm 操作导致的（`helm install`、`helm upgrade` 或 `helm uninstall` 由 HelmChart CR 执行）。 + +:::note + +HelmRelease CR 会发出 Kubernetes 事件，用于检测底层 Helm 版本修改并将其锁定回原位。要查看这些事件，你可以使用 `kubectl describe helmrelease -n `。你还可以查看此 Operator 的日志，了解检测到更改的时间以及哪些资源被尝试更改。 + +::: + +这两种资源都是为 Operator / System 命名空间中的所有 Helm Chart 创建的，用于避免低权限用户的权限升级。 + +### 高级 Helm Project Operator 配置 + +有关高级配置的更多信息，请参阅[此页面](https://github.com/rancher/prometheus-federator/blob/main/charts/prometheus-federator/0.0.1/README.md#advanced-helm-project-operator-configuration)。 + + + +### Local 集群上的 Prometheus Federator + +Prometheus Federator 是一个资源密集型应用程序。你可以将其安装到 Local 集群（**不推荐**）。 + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md new file mode 100644 index 00000000000..442efff377e --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-manager-architecture/rancher-manager-architecture.md @@ -0,0 +1,21 @@ +--- +title: 架构 +--- + + + + + +本章节重点介绍 [Rancher Server 及其组件](rancher-server-and-components.md) 以及 [Rancher 如何与下游 Kubernetes 集群通信](communicating-with-downstream-user-clusters.md)。 + +有关安装 Rancher 的不同方式的信息，请参见[安装选项概述](../../getting-started/installation-and-upgrade/installation-and-upgrade.md#安装方式概述)。 + +有关 Rancher API Server 的主要功能，请参见[概述](../../getting-started/overview.md#rancher-api-server-的功能)。 + +有关如何为 Rancher Server 设置底层基础架构，请参见[架构推荐](architecture-recommendations.md)。 + +:::note + +本节默认你已对 Docker 和 Kubernetes 有一定的了解。如果你需要了解 Kubernetes 组件如何协作，请参见 [Kubernetes 概念](../kubernetes-concepts.md)。 + +::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/hardening-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/hardening-guides.md new file mode 100644 index 00000000000..6de7d34574f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/hardening-guides.md @@ -0,0 +1,55 @@ +--- +title: Rancher 自我评估和加固指南 +--- + + + + + +Rancher 为每个受支持的 Rancher 版本的 Kubernetes 发行版提供了特定的安全强化指南。 + +## Rancher Kubernetes 发行版 + +Rancher 使用以下 Kubernetes 发行版： + +- [**RKE**](https://rancher.com/docs/rke/latest/en/)（Rancher Kubernetes Engine）是经过 CNCF 认证的 Kubernetes 发行版，完全在 Docker 容器中运行。 +- [**RKE2**](https://docs.rke2.io/) 是一个完全合规的 Kubernetes 发行版，专注于安全和合规性。 +- [**K3s**](https://docs.k3s.io/) 是一个完全合规的，轻量级 Kubernetes 发行版。它易于安装，内存需求只有上游 Kubernetes 的一半，所有组件都在一个小于 100 MB 的二进制文件中。 + +要加固运行未列出的发行版的 Kubernetes 集群，请参阅 Kubernetes 提供商文档。 + +## 加固指南和 Benchmark 版本 + +每个自我评估指南都附有强化指南。这些指南与列出的 Rancher 版本一起进行了测试。每个自我评估指南都在特定的 Kubernetes 版本和 CIS Benchmark 版本上进行了测试。如果 CIS Benchmark 尚未针对你的 Kubernetes 版本进行验证，你可以使用现有指南，直到添加适合你的版本的指南。 + +### RKE 指南 + +| Kubernetes 版本 | CIS Benchmark 版本 | 自我评估指南 | 加固指南 | +|--------------------|-----------------------|-----------------------|------------------| +| Kubernetes v1.23 | CIS v1.23 | [链接](rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md) | [链接](rke1-hardening-guide/rke1-hardening-guide.md) | +| Kubernetes v1.24 | CIS v1.24 | [链接](rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md) | [链接](rke1-hardening-guide/rke1-hardening-guide.md) | +| Kubernetes v1.25/v1.26/v1.27 | CIS v1.7 | [链接](rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md) | [链接](rke1-hardening-guide/rke1-hardening-guide.md) | + +### RKE2 指南 + +| 类型 | Kubernetes 版本 | CIS Benchmark 版本 | 自我评估指南 | 加固指南 | +|------|--------------------|-----------------------|-----------------------|------------------| +| Rancher provisioned RKE2 | Kubernetes v1.23 | CIS v1.23 | [链接](rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md) | [链接](rke2-hardening-guide/rke2-hardening-guide.md) | +| Rancher provisioned RKE2 | Kubernetes v1.24 | CIS v1.24 | [链接](rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md) | [链接](rke2-hardening-guide/rke2-hardening-guide.md) | +| Rancher provisioned RKE2 | Kubernetes v1.25/v1.26/v1.27 | CIS v1.7 | [链接](rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md) | [链接](rke2-hardening-guide/rke2-hardening-guide.md) | +| Standalone RKE2 | Kubernetes v1.25/v1.26/v1.27 | CIS v1.7 | [链接](https://docs.rke2.io/security/cis_self_assessment123) | [链接](https://docs.rke2.io/security/hardening_guide) | + +### K3s 指南 + +| 类型 | Kubernetes 版本 | CIS Benchmark 版本 | 自我评估指南 | 加固指南 | +|------|--------------------|-----------------------|-----------------------|------------------| +| Rancher provisioned K3s cluster | Kubernetes v1.23 | CIS v1.23 | [链接](k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md) | [链接](k3s-hardening-guide/k3s-hardening-guide.md) | +| Rancher provisioned K3s cluster | Kubernetes v1.24 | CIS v1.24 | [链接](k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md) | [链接](k3s-hardening-guide/k3s-hardening-guide.md) | +| Rancher provisioned K3s cluster | Kubernetes v1.25/v1.26/v1.27 | CIS v1.7 | [链接](k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md) | [链接](k3s-hardening-guide/k3s-hardening-guide.md) | +| Standalone K3s | Kubernetes v1.22 up to v1.24 | CIS v1.23 | [链接](https://docs.k3s.io/security/self-assessment) | [链接](https://docs.k3s.io/security/hardening-guide) | + +## 在 SELinux 上使用 Rancher + +[Security-Enhanced Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) 是一个内核模块，为 Linux 添加了额外的访问控制和安全工具。SELinux 过去曾被政府机构使用，现在已成为行业标准。SELinux 在 RHEL 和 CentOS 上默认启用。 + +要将 Rancher 与 SELinux 结合使用，我们建议[安装](../selinux-rpm/about-rancher-selinux.md) `rancher-selinux`。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md new file mode 100644 index 00000000000..d40edd47e96 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-hardening-guide.md @@ -0,0 +1,744 @@ +--- +title: K3s 加固指南 +--- + + + + + +本文档提供了针对生产环境的 K3s 集群进行加固的具体指导，以便在使用 Rancher 部署之前进行配置。它概述了满足信息安全中心（Center for Information Security, CIS）Kubernetes benchmark controls 所需的配置和控制。 + +:::note +这份加固指南描述了如何确保你集群中的节点安全。我们建议你在安装 Kubernetes 之前遵循本指南。 +::: + +此加固指南适用于 K3s 集群，并与以下版本的 CIS Kubernetes Benchmark、Kubernetes 和 Rancher 相关联： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|------------------------------| +| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 至 v1.26 | + +:::note +在 Benchmark v1.7 中，不再需要 `--protect-kernel-defaults` (4.2.6) 参数，并已被 CIS 删除。 +::: + +有关如何评估加固的 K3s 集群与官方 CIS benchmark 的更多细节，请参考特定 Kubernetes 和 CIS benchmark 版本的 K3s 自我评估指南。 + +K3s 在不需要修改的情况下通过了许多 Kubernetes CIS controls，因为它默认应用了几个安全缓解措施。然而，有一些值得注意的例外情况，需要手动干预才能完全符合 CIS Benchmark 要求： + +1. K3s 不修改主机操作系统。任何主机级别的修改都需要手动完成。 +2. 某些 CIS policy controls，例如 `NetworkPolicies` 和 `PodSecurityStandards`（在 v1.24 及更早版本中为 `PodSecurityPolicies`），会限制集群功能。 + 你必须选择让 K3s 配置这些策略。在你的命令行标志或配置文件中添加相应的选项（启用准入插件），并手动应用适当的策略。 + 请参阅以下详细信息。 + +CIS Benchmark 的第一部分（1.1）主要关注于 Pod manifest 的权限和所有权。由于发行版中的所有内容都打包在一个二进制文件中，因此这一部分不适用于 K3s 的核心组件。 + +## 主机级别要求 + +### 确保 `protect-kernel-defaults`已经设置 + + + + +自 CIS benchmark 1.7 开始，不再需要`protect-kernel-defaults`。 + + + + +这是一个 kubelet 标志，如果所需的内核参数未设置或设置为与 kubelet 的默认值不同的值，将导致 kubelet 退出。 + +可以在 Rancher 的集群配置中设置 `protect-kernel-defaults` 标志。 + +```yaml +spec: + rkeConfig: + machineSelectorConfig: + - config: + protect-kernel-defaults: true +``` + + + + +### 设置内核参数 + +建议为集群中的所有节点类型设置以下 `sysctl` 配置。在 `/etc/sysctl.d/90-kubelet.conf` 中设置以下参数： + +```ini +vm.panic_on_oom=0 +vm.overcommit_memory=1 +kernel.panic=10 +kernel.panic_on_oops=1 +``` + +运行 `sudo sysctl -p /etc/sysctl.d/90-kubelet.conf` 以启用设置。 + +此配置需要在设置 kubelet 标志之前完成，否则 K3s 将无法启动。 + +## Kubernetes 运行时要求 + +CIS Benchmark 的运行时要求主要围绕 Pod 安全（通过 PSP 或 PSA）、网络策略和 API 服务器审计日志展开。 + +默认情况下，K3s 不包含任何 Pod 安全或网络策略。然而，K3s 附带一个控制器，可以强制执行你创建的任何网络策略。默认情况下，K3s 启用了 `PodSecurity` 和 `NodeRestriction` 等多个准入控制器。 + +### Pod 安全 + + + + +K3s v1.25 及更新版本支持 [Pod 安全准入(PSA)](https://kubernetes.io/docs/concepts/security/pod-security-admission/)，用于控制 Pod 安全性。 + +你可以在 Rancher 中通过集群配置，设置 `defaultPodSecurityAdmissionConfigurationTemplateName` 字段来指定 PSA 配置： + +```yaml +spec: + defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted +``` + +Rancher 提供了 `rancher-restricted` 模板，用于强制执行高度限制性的 Kubernetes 上游 [`Restricted`](https://kubernetes.io/docs/concepts/security/pod-security-standards/#restricted) 配置文件，其中包含了 Pod 加固的最佳实践。 + + + + +K3s v1.24 及更早版本支持 [Pod 安全策略 (PSP)](https://v1-24.docs.kubernetes.io/docs/concepts/security/pod-security-policy/) 以控制 Pod 安全性。 + +你可以在 Rancher 中通过集群配置，传递以下标志来启用 PSPs： + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount +``` + +这会保留 `NodeRestriction` 插件并启用 `PodSecurityPolicy`。 + +启用 PSPs 后，你可以应用策略来满足 CIS Benchmark 第 5.2 节中描述的必要控制。 + +:::note +这些是 CIS Benchmark 中的手动检查。CIS 扫描结果将标记为 `warning`，因为需要集群操作员进行手动检查。 +::: + +以下是合规的 PSP 示例： + +```yaml +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: restricted-psp +spec: + privileged: false # CIS - 5.2.1 + allowPrivilegeEscalation: false # CIS - 5.2.5 + requiredDropCapabilities: # CIS - 5.2.7/8/9 + - ALL + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'downwardAPI' + - 'csi' + - 'persistentVolumeClaim' + - 'ephemeral' + hostNetwork: false # CIS - 5.2.4 + hostIPC: false # CIS - 5.2.3 + hostPID: false # CIS - 5.2.2 + runAsUser: + rule: 'MustRunAsNonRoot' # CIS - 5.2.6 + seLinux: + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + - min: 1 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + - min: 1 + max: 65535 + readOnlyRootFilesystem: false +``` + +要使示例 PSP 生效，我们需要创建一个 `ClusterRole` 和 一个`ClusterRoleBinding`。我们还需要为需要额外权限的系统级 Pod 提供“系统无限制策略”，以及允许必要的 sysctls 来实现 ServiceLB 完整功能的额外策略。 + +```yaml +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: restricted-psp +spec: + privileged: false + allowPrivilegeEscalation: false + requiredDropCapabilities: + - ALL + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'downwardAPI' + - 'csi' + - 'persistentVolumeClaim' + - 'ephemeral' + hostNetwork: false + hostIPC: false + hostPID: false + runAsUser: + rule: 'MustRunAsNonRoot' + seLinux: + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + - min: 1 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + - min: 1 + max: 65535 + readOnlyRootFilesystem: false +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: system-unrestricted-psp + annotations: + seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*' +spec: + allowPrivilegeEscalation: true + allowedCapabilities: + - '*' + fsGroup: + rule: RunAsAny + hostIPC: true + hostNetwork: true + hostPID: true + hostPorts: + - max: 65535 + min: 0 + privileged: true + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + volumes: + - '*' +--- +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: svclb-psp + annotations: + seccomp.security.alpha.kubernetes.io/allowedProfileNames: '*' +spec: + allowPrivilegeEscalation: false + allowedCapabilities: + - NET_ADMIN + allowedUnsafeSysctls: + - net.ipv4.ip_forward + - net.ipv6.conf.all.forwarding + fsGroup: + rule: RunAsAny + hostPorts: + - max: 65535 + min: 0 + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:restricted-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + verbs: + - use + resourceNames: + - restricted-psp +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:system-unrestricted-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - system-unrestricted-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:svclb-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - svclb-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:svc-local-path-provisioner-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - system-unrestricted-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:svc-coredns-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - system-unrestricted-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: psp:svc-cis-operator-psp +rules: +- apiGroups: + - policy + resources: + - podsecuritypolicies + resourceNames: + - system-unrestricted-psp + verbs: + - use +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: default:restricted-psp +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted-psp +subjects: +- kind: Group + name: system:authenticated + apiGroup: rbac.authorization.k8s.io +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: system-unrestricted-node-psp-rolebinding +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:system-unrestricted-psp +subjects: +- apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:nodes +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: system-unrestricted-svc-acct-psp-rolebinding + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:system-unrestricted-psp +subjects: +- apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: svclb-psp-rolebinding + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:svclb-psp +subjects: +- kind: ServiceAccount + name: svclb +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: svc-local-path-provisioner-psp-rolebinding + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:svc-local-path-provisioner-psp +subjects: +- kind: ServiceAccount + name: local-path-provisioner-service-account +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: svc-coredns-psp-rolebinding + namespace: kube-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:svc-coredns-psp +subjects: +- kind: ServiceAccount + name: coredns +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: svc-cis-operator-psp-rolebinding + namespace: cis-operator-system +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:svc-cis-operator-psp +subjects: +- kind: ServiceAccount + name: cis-operator-serviceaccount +``` + +上述策略可以放置在 `/var/lib/rancher/k3s/server/manifests` 目录下名为 `policy.yaml` 的文件中。在启动 K3s 之前，必须创建策略文件和其目录结构。建议限制访问权限以避免泄露潜在的敏感信息。 + +```shell +sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/manifests +``` + +:::note +CNI、DNS 和 Ingress 等关键 Kubernetes 组件在 `kube-system` 命名空间中作为 Pod 运行。因此，这个命名空间的限制政策较少，从而使这些组件能够正常运行。 +::: + + + + +### 网络策略 + +CIS 要求所有命名空间应用网络策略，合理限制进入命名空间和 Pod 的流量。 + +:::note +这些是 CIS Benchmark 中的手动检查。CIS 扫描结果将标记为 `warning`，因为需要集群操作员进行手动检查。 +::: + +网络策略可以放置在 `/var/lib/rancher/k3s/server/manifests` 目录下的 `policy.yaml` 文件中。如果该目录不是作为 PSP（如上所述）的一部分创建的，则必须首先创建该目录。 + +```shell +sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/manifests +``` + +以下是合规的网络策略示例： + +```yaml +--- +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: intra-namespace + namespace: kube-system +spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + name: kube-system +--- +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: intra-namespace + namespace: default +spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + name: default +--- +kind: NetworkPolicy +apiVersion: networking.k8s.io/v1 +metadata: + name: intra-namespace + namespace: kube-public +spec: + podSelector: {} + ingress: + - from: + - namespaceSelector: + matchLabels: + name: kube-public +``` + +除非特意允许，否则活动限制会阻止 DNS。以下是允许 DNS 相关流量的网络策略示例： + +```yaml +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-network-dns-policy + namespace: +spec: + ingress: + - ports: + - port: 53 + protocol: TCP + - port: 53 + protocol: UDP + podSelector: + matchLabels: + k8s-app: kube-dns + policyTypes: + - Ingress +``` + +如果没有创建网络策略来允许访问，则默认情况下会阻止 metrics-server 和 Traefik Ingress 控制器。 + +```yaml +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-all-metrics-server + namespace: kube-system +spec: + podSelector: + matchLabels: + k8s-app: metrics-server + ingress: + - {} + policyTypes: + - Ingress +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-all-svclbtraefik-ingress + namespace: kube-system +spec: + podSelector: + matchLabels: + svccontroller.k3s.cattle.io/svcname: traefik + ingress: + - {} + policyTypes: + - Ingress +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: allow-all-traefik-v121-ingress + namespace: kube-system +spec: + podSelector: + matchLabels: + app.kubernetes.io/name: traefik + ingress: + - {} + policyTypes: + - Ingress +``` + +:::note +你必须像平常一样管理你创建的任何其他命名空间的网络策略。 +::: + +### API server 审计配置 + +CIS 要求 1.2.19 至 1.2.22 与配置 API server 审核日志相关。默认情况下，K3s 不会创建日志目录和审计策略，因为每个用户的审计策略要求和环境都是特定的。 + +如果你需要日志目录，则必须在启动 K3s 之前创建它。我们建议限制访问权限以避免泄露敏感信息。 + +```bash +sudo mkdir -p -m 700 /var/lib/rancher/k3s/server/logs +``` + +以下是用于记录请求元数据的初始审计策略。应将策略写入到 `/var/lib/rancher/k3s/server` 目录下名为 `audit.yaml` 的文件中。有关 API server 的策略配置的详细信息，请参阅 [官方 Kubernetes 文档](https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/)。 + +```yaml +--- +apiVersion: audit.k8s.io/v1 +kind: Policy +rules: +- level: Metadata +``` + +还需要进一步配置才能通过 CIS 检查。这些在 K3s 中默认不配置，因为它们根据你的环境和需求而有所不同： + +- 确保 `--audit-log-path` 参数已经设置。 +- 确保 `--audit-log-maxage` 参数设置为 30 或适当的值。 +- 确保 `--audit-log-maxbackup` 参数设置为 10 或适当的值。 +- 确保 `--audit-log-maxsize` 参数设置为 100 或适当的值。 + +综合起来，要启用和配置审计日志，请将以下行添加到 Rancher 的 K3s 集群配置文件中： + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS 3.2.1 + - audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS 1.2.18 + - audit-log-maxage=30 # CIS 1.2.19 + - audit-log-maxbackup=10 # CIS 1.2.20 + - audit-log-maxsize=100 # CIS 1.2.21 +``` + +### Controller Manager 要求 + +CIS 要求 1.3.1 检查 Controller Manager 中的垃圾收集设置。垃圾收集对于确保资源充足可用性并避免性能和可用性下降非常重要。根据你的系统资源和测试结果，选择一个适当的阈值来激活垃圾收集。 + +你可以在 Rancher 的 K3s 集群文件中设置以下配置来解决此问题。下面的值仅是一个示例，请根据当前环境设置适当的阈值。 + +```yaml +spec: + rkeConfig: + machineGlobalConfig: + kube-controller-manager-arg: + - terminated-pod-gc-threshold=10 # CIS 1.3.1 +``` + +### 配置 `default` Service Account + +Kubernetes 提供了一个名为 `default` 的 service account，供集群工作负载使用，其中没有为 Pod 分配特定的 service account。当 Pod 需要从 Kubernetes API 获取访问权限时，应为该 Pod 创建一个特定的 service account，并为该 service account 授予权限。 + +对于 CIS 5.1.5，`default` service account 应配置为不提供 service account 令牌，并且不具有任何明确的权限分配。 + +可以通过在每个命名空间中将 `default` service account 的 `automountServiceAccountToken` 字段更新为 `false` 来解决此问题。 + +对于内置命名空间（`kube-system`、`kube-public`、`kube-node-lease` 和 `default`）中的 `default` service accounts，K3s 不会自动执行此操作。 + +将以下配置保存到名为 `account_update.yaml` 的文件中。 + +```yaml +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: default +automountServiceAccountToken: false +``` + +创建一个名为 `account_update.sh` 的 Bash 脚本文件。确保使用 `chmod +x account_update.sh` 给脚本添加可执行权限。 + +```shell +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" +done +``` + +每次向你的集群添加新的 service account 时，运行该脚本。 + +## 加固版 K3s 模板配置参考 + +Rancher 使用以下参考模板配置，基于本指南中的每个 CIS 控件创建加固过的自定义 K3s 集群。此参考内容不包括其他必需的**集群配置**指令，这些指令因你的环境而异。 + + + + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: # 定义集群名称 +spec: + defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted + enableNetworkPolicy: true + kubernetesVersion: # 定义 K3s 版本 + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - enable-admission-plugins=NodeRestriction,ServiceAccount # CIS 1.2.15, 1.2.13 + - audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS 3.2.1 + - audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS 1.2.18 + - audit-log-maxage=30 # CIS 1.2.19 + - audit-log-maxbackup=10 # CIS 1.2.20 + - audit-log-maxsize=100 # CIS 1.2.21 + - request-timeout=300s # CIS 1.2.22 + - service-account-lookup=true # CIS 1.2.24 + kube-controller-manager-arg: + - terminated-pod-gc-threshold=10 # CIS 1.3.1 + secrets-encryption: true + machineSelectorConfig: + - config: + kubelet-arg: + - make-iptables-util-chains=true # CIS 4.2.7 +``` + + + + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: # 定义集群名称 +spec: + enableNetworkPolicy: true + kubernetesVersion: # 定义 K3s 版本 + rkeConfig: + machineGlobalConfig: + kube-apiserver-arg: + - enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount # CIS 1.2.15, 5.2, 1.2.13 + - audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml # CIS 3.2.1 + - audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log # CIS 1.2.18 + - audit-log-maxage=30 # CIS 1.2.19 + - audit-log-maxbackup=10 # CIS 1.2.20 + - audit-log-maxsize=100 # CIS 1.2.21 + - request-timeout=300s # CIS 1.2.22 + - service-account-lookup=true # CIS 1.2.24 + kube-controller-manager-arg: + - terminated-pod-gc-threshold=10 # CIS 1.3.1 + secrets-encryption: true + machineSelectorConfig: + - config: + kubelet-arg: + - make-iptables-util-chains=true # CIS 4.2.7 + protect-kernel-defaults: true # CIS 4.2.6 +``` + + + + +## 结论 + +如果你按照本指南操作，由 Rancher 提供的 K3s 自定义集群将配置为通过 CIS Kubernetes Benchmark 测试。你可以查看我们的 K3s 自我评估指南，了解我们是如何验证每个 benchmarks 的，并且你可以在你的集群上执行相同的操作。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md index ddf9809440a..5bfb97c99f7 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md @@ -1,36 +1,40 @@ --- -title: K3s Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.23 +title: K3s 自我评估指南 - CIS Benchmark v1.23 - K8s v1.23 --- -This document is a companion to the [K3s Hardening Guide](../../../../pages-for-subheaders/k3s-hardening-guide.md), which provides prescriptive guidance on how to harden K3s clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. + + + -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: +本文档是 [K3s 加固指南](k3s-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 K3s 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 -| Rancher Version | CIS Benchmark Version | Kubernetes Version | +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | |-----------------|-----------------------|--------------------| | Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | -This document is for Rancher operators, security teams, auditors and decision makers. +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.23 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 -## Testing Methodology +## 测试方法 -Each control in the CIS Kubernetes Benchmark was evaluated against a K3s cluster that was configured according to the accompanying hardening guide. +每个 CIS Kubernetes Benchmark 中的 control 都根据附带的加固指南评估了针对 K3s 集群的配置。 -Where control audits differ from the original CIS benchmark, the audit commands specific to K3s are provided for testing. +当 control 审计与原始的 CIS benchmark 不同的时候，提供了针对 K3s 的特定审计命令，以供测试使用。 -These are the possible results for each control: +以下是每个 control 可能的结果： -- **Pass** - The K3s cluster passes the audit outlined in the benchmark. -- **Not Applicable** - The control is not applicable to K3s because of how it is designed to operate. The remediation section explains why. -- **Warn** - The control is manual in the CIS benchmark and it depends on the cluster's use-case or some other factor that must be determined by the cluster operator. These controls have been evaluated to ensure K3s doesn't prevent their implementation, but no further configuration or auditing of the cluster has been performed. +- **Pass（通过）** - K3s 集群通过了 benchmark 中概述的审计。 +- **Not Applicable（不适用）** - 由于 K3s 的设计方式，该 control 不适用于 K3s。在补救措施部分解释了原因。 +- **Warn（警告）** - 在 CIS benchmark 中，该 control 是手动的，它取决于集群的使用情况或其他必须由集群操作员确定的因素。这些 control 措施已经过评估，以确保 K3s 不会阻止其实施，但尚未对集群进行进一步的配置或审计。 -This guide makes the assumption that K3s is running as a Systemd unit. Your installation may vary. Adjust the "audit" commands to fit your scenario. +本指南假设 K3s 作为 Systemd 单元运行。你的安装可能会有所不同。调整"审计"命令以适合你的场景。 :::note -This guide only covers `automated` (previously called `scored`) tests. +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md new file mode 100644 index 00000000000..464073a825f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md @@ -0,0 +1,3208 @@ +--- +title: K3s 自我评估指南 - CIS Benchmark v1.24 - K8s v1.24 +--- + + + + + +本文档是 [K3s 加固指南](k3s-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 K3s 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.24 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +每个 CIS Kubernetes Benchmark 中的 control 都根据附带的加固指南评估了针对 K3s 集群的配置。 + +当 control 审计与原始的 CIS benchmark 不同的时候，提供了针对 K3s 的特定审计命令，以供测试使用。 + +以下是每个 control 可能的结果： + +- **Pass（通过）** - K3s 集群通过了 benchmark 中概述的审计。 +- **Not Applicable（不适用）** - 由于 K3s 的设计方式，该 control 不适用于 K3s。在补救措施部分解释了原因。 +- **Warn（警告）** - 在 CIS benchmark 中，该 control 是手动的，它取决于集群的使用情况或其他必须由集群操作员确定的因素。这些 control 措施已经过评估，以确保 K3s 不会阻止其实施，但尚未对集群进行进一步的配置或审计。 + +本指南假设 K3s 作为 Systemd 单元运行。你的安装可能会有所不同。调整"审计"命令以适合你的场景。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 644 /etc/kubernetes/manifests/kube-apiserver.yaml + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-controller-manager.yaml + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-scheduler.yaml + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /etc/kubernetes/manifests/etcd.yaml + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /etc/kubernetes/manifests/etcd.yaml + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.1.11 +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** Not Applicable + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 scheduler + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root scheduler + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 controllermanager + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/controller.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root controllermanager + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/tls +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit:** + +```bash +find /var/lib/rancher/k3s/server/tls | xargs stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root root:root +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.crt + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.key + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' is present OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 410 1 0 Sep11 ? 00:01:50 /usr/bin/containerd root 539 1 0 Sep11 ? 00:00:09 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 45213 45195 3 Sep11 ? 00:45:14 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 47188 1 0 Sep11 ? 00:01:00 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 02b8fdd94b7628d575ace92f337dcc93202151d07f33858341cfcb0178fea586 -address /run/k3s/containerd/containerd.sock root 47235 1 0 Sep11 ? 00:00:33 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id eb0f03c4bc1125ae34e5de77c7eaeb9c2c1e46bec0bb106b99281df7353f9ded -address /run/k3s/containerd/containerd.sock root 47948 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5eece7110bd3876414f890ed1b7158eb91f60d1f14450a9f8c77a0fc62b73c4a -address /run/k3s/containerd/containerd.sock root 48047 1 0 Sep11 ? 00:00:32 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4ef831f4bb8830a0a7b46a1e8be58197be00846ea0caa094014ea7ce3adf008e -address /run/k3s/containerd/containerd.sock root 48220 1 0 Sep11 ? 00:00:33 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id f8d5d3791ad2b1a8534476aa2fbdf8da1e4f0b9fdcdd26b1ae161142458a5336 -address /run/k3s/containerd/containerd.sock root 48878 1 0 Sep11 ? 00:00:32 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 41b93a6581e272b0852257b3d7c44ab4cadcdf465a3579a8b29b493ea75feaf7 -address /run/k3s/containerd/containerd.sock root 49870 1 0 Sep11 ? 00:00:30 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id be329cdccda4049f048c759a2b1915cd72526473ff7768e895bcfc153a66c125 -address /run/k3s/containerd/containerd.sock root 50271 1 0 Sep11 ? 00:00:32 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id cceb5d26eba7d7d3c474a2dc842b72495689f4e40055505ea2598e4ac6849d40 -address /run/k3s/containerd/containerd.sock root 50571 1 0 Sep11 ? 00:00:32 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3a26e691e0d0186165d523bef22b0af71f175c39df82845b32ce7b4e3d652d4a -address /run/k3s/containerd/containerd.sock root 96693 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9ed77e6e58410f1295f730ba4a8f8daf7aff8f4e896e24219a8fa710729c2287 -address /run/k3s/containerd/containerd.sock root 97726 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 1be4fe50e6bb29d4549f8b03102b5dc15b7669fe56b32beae3f701cf6d2e58cc -address /run/k3s/containerd/containerd.sock root 97923 1 1 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/484bf694b486d93cc93bcf90f74d5c77628550d4456f21760fd720b88c93881e/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7c65c47376f36a8980a5bebad7ffdc1abe1853821cf6c4a1d45f1b6be3452062 -address /run/k3s/containerd/containerd.sock +``` + +### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --kubelet-https parameter. + +### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'ServiceAccount' +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.18 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-path' +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxage' +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxbackup' +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-log-maxsize' +``` + +**Expected Result**: + +```console +'--audit-log-maxsize' is greater or equal to 100 +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.23 Ensure that the --request-timeout argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, --request-timeout=300s + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'request-timeout' +``` + +**Expected Result**: + +```console +'--request-timeout' is not present OR '--request-timeout' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-lookup' +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.25 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'service-account-key-file' +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.2.29 +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'encryption-provider-config' +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.31 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit:** + +```bash +grep aescbc /path/to/encryption-config.json +``` + +### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'bind-address' +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.1 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.2 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.3 +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.4 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.5 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.6 +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Manual) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.7 +``` + +**Expected Result**: + +```console +'trusted-ca-file' is present +``` + +**Returned Value**: + +```console +trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Manual) + + +**Result:** warn + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +**Returned Value**: + +```console +600 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'600' is equal to '600' +``` + +**Returned Value**: + +```console +600 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +**Returned Value**: + +```console +644 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/kubelet/config.yaml + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" | grep -v grep; else echo "--anonymous-auth=false"; fi' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" | grep -v grep; else echo "--authorization-mode=Webhook"; fi' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" | grep -v grep; else echo "--client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt"; fi' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:31:58 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:31:58Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:32:02 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:32:02Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-3-32 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=cc5faddd-9c87-4a04-97c2-dcf4cbcbe10f --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' +``` + +### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'protect-kernel-defaults' +``` + +**Expected Result**: + +```console +'--protect-kernel-defaults' is equal to 'true' +``` + +**Returned Value**: + +```console +Sep 11 20:32:02 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:32:02Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-3-32 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=cc5faddd-9c87-4a04-97c2-dcf4cbcbe10f --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' +``` + +**Expected Result**: + +```console +'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:32:02 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:32:02Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-3-32 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=cc5faddd-9c87-4a04-97c2-dcf4cbcbe10f --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +### 4.2.9 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'--event-qps' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 410 1 0 Sep11 ? 00:01:50 /usr/bin/containerd root 45213 45195 3 Sep11 ? 00:45:15 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:32:02 ip-172-31-3-32 k3s[45195]: time="2023-09-11T20:32:02Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-3-32 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=cc5faddd-9c87-4a04-97c2-dcf4cbcbe10f --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 410 1 0 Sep11 ? 00:01:50 /usr/bin/containerd root 45213 45195 3 Sep11 ? 00:45:15 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 410 1 0 Sep11 ? 00:01:50 /usr/bin/containerd root 45213 45195 3 Sep11 ? 00:45:15 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** warn + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +### 5.2.7 Minimize the admission of root containers (Automated) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Automated) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md new file mode 100644 index 00000000000..8443a80b530 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/k3s-hardening-guide/k3s-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md @@ -0,0 +1,3215 @@ +--- +title: K3s 自我评估指南 - CIS Benchmark v1.7 - K8s v1.25/v1.26/v1.27 +--- + + + + + +本文档是 [K3s 加固指南](k3s-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 K3s 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25/v1.26/v1.27 | + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.7 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +每个 CIS Kubernetes Benchmark 中的 control 都根据附带的加固指南评估了针对 K3s 集群的配置。 + +当 control 审计与原始的 CIS benchmark 不同的时候，提供了针对 K3s 的特定审计命令，以供测试使用。 + +以下是每个 control 可能的结果： + +- **Pass（通过）** - K3s 集群通过了 benchmark 中概述的审计。 +- **Not Applicable（不适用）** - 由于 K3s 的设计方式，该 control 不适用于 K3s。在补救措施部分解释了原因。 +- **Warn（警告）** - 在 CIS benchmark 中，该 control 是手动的，它取决于集群的使用情况或其他必须由集群操作员确定的因素。这些 control 措施已经过评估，以确保 K3s 不会阻止其实施，但尚未对集群进行进一步的配置或审计。 + +本指南假设 K3s 作为 Systemd 单元运行。你的安装可能会有所不同。调整"审计"命令以适合你的场景。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-apiserver.yaml +Not Applicable. + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml +Not Applicable. + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-controller-manager.yaml +Not Applicable. + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml +Not Applicable. + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-scheduler.yaml +Not Applicable. + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml +Not Applicable. + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /etc/kubernetes/manifests/etcd.yaml +Not Applicable. + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /etc/kubernetes/manifests/etcd.yaml +Not Applicable. + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 +Not Applicable. + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root +Not Applicable. + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.1.11 +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** Not Applicable + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd +Not Applicable. + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 scheduler + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root scheduler + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 controllermanager + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/controller.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root controllermanager + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/cred/controller.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /var/lib/rancher/k3s/server/tls + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/tls +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.crt + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.crt +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/pki/*.key + +**Audit:** + +```bash +stat -c %n %a /var/lib/rancher/k3s/server/tls/*.key +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'anonymous-auth' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 663 1 0 Sep11 ? 00:00:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 3021 1 0 Sep11 ? 00:00:29 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4790d392966915d995e666002c56ed4cce6dd86f305ce2ee390547a1fcbf6c82 -address /run/k3s/containerd/containerd.sock root 3035 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9d73e6a160ccde6c7c7d4ba0df3b9f696e3de2ebfc0f19a1dcbdf13aea496427 -address /run/k3s/containerd/containerd.sock root 3235 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 587f6221ee9f36c877231ada8d816a799dfda186332c33378d1eb16c72cdc87d -address /run/k3s/containerd/containerd.sock root 4435 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a74c9ec7d99785c2f2d4e6826aa80c22eb8b38249e8f99679ece00a818e9b7b3 -address /run/k3s/containerd/containerd.sock root 4985 1 0 Sep11 ? 00:00:53 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b0c9784dbe0fcbe8be10c857976e70ec84a208cb814e87b5ca085a02d434f8c -address /run/k3s/containerd/containerd.sock root 5056 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id f3231ff35f18056e74eda14907f296be15f7ea1c6ae5ab7904e27d4d18183301 -address /run/k3s/containerd/containerd.sock root 5868 1 0 Sep11 ? 00:00:27 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3e908c4d0b10df275bdef6f72fbcfa09517d11cf749236ad020364e14d77bc93 -address /run/k3s/containerd/containerd.sock root 6158 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id bec5780a5c73fa3154aa3b5ee26cdf23202db821205893e7e66ae17e6103e97b -address /run/k3s/containerd/containerd.sock root 7366 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a81b78845bdcaef710314c93e5ea0d0617f37a8929472f7b570ab90c6667f57f -address /run/k3s/containerd/containerd.sock root 97274 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id c90652c935d9e79af45b9a9ac2b4fe315e2a761a0525737fdb95c680123a164c -address /run/k3s/containerd/containerd.sock root 98309 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 431a2763636488efd7d104c228d41f4f2ecd8c06a7fc375d8977ab0d238936a8 -address /run/k3s/containerd/containerd.sock root 98493 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b480d2fa55c8cc0105ec1902887040c017f03d5b1eb6c73c24bb9d523ad9b37 -address /run/k3s/containerd/containerd.sock +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' is present OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 663 1 0 Sep11 ? 00:00:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 3021 1 0 Sep11 ? 00:00:29 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4790d392966915d995e666002c56ed4cce6dd86f305ce2ee390547a1fcbf6c82 -address /run/k3s/containerd/containerd.sock root 3035 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9d73e6a160ccde6c7c7d4ba0df3b9f696e3de2ebfc0f19a1dcbdf13aea496427 -address /run/k3s/containerd/containerd.sock root 3235 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 587f6221ee9f36c877231ada8d816a799dfda186332c33378d1eb16c72cdc87d -address /run/k3s/containerd/containerd.sock root 4435 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a74c9ec7d99785c2f2d4e6826aa80c22eb8b38249e8f99679ece00a818e9b7b3 -address /run/k3s/containerd/containerd.sock root 4985 1 0 Sep11 ? 00:00:53 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b0c9784dbe0fcbe8be10c857976e70ec84a208cb814e87b5ca085a02d434f8c -address /run/k3s/containerd/containerd.sock root 5056 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id f3231ff35f18056e74eda14907f296be15f7ea1c6ae5ab7904e27d4d18183301 -address /run/k3s/containerd/containerd.sock root 5868 1 0 Sep11 ? 00:00:27 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3e908c4d0b10df275bdef6f72fbcfa09517d11cf749236ad020364e14d77bc93 -address /run/k3s/containerd/containerd.sock root 6158 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id bec5780a5c73fa3154aa3b5ee26cdf23202db821205893e7e66ae17e6103e97b -address /run/k3s/containerd/containerd.sock root 7366 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a81b78845bdcaef710314c93e5ea0d0617f37a8929472f7b570ab90c6667f57f -address /run/k3s/containerd/containerd.sock root 97274 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id c90652c935d9e79af45b9a9ac2b4fe315e2a761a0525737fdb95c680123a164c -address /run/k3s/containerd/containerd.sock root 98309 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 431a2763636488efd7d104c228d41f4f2ecd8c06a7fc375d8977ab0d238936a8 -address /run/k3s/containerd/containerd.sock root 98493 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b480d2fa55c8cc0105ec1902887040c017f03d5b1eb6c73c24bb9d523ad9b37 -address /run/k3s/containerd/containerd.sock +``` + +### 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'kubelet-certificate-authority' +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= +Permissive - When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. + +### 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.7 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'authorization-mode' +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.9 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' is present +``` + +**Returned Value**: + +```console +root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 663 1 0 Sep11 ? 00:00:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 3021 1 0 Sep11 ? 00:00:29 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4790d392966915d995e666002c56ed4cce6dd86f305ce2ee390547a1fcbf6c82 -address /run/k3s/containerd/containerd.sock root 3035 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9d73e6a160ccde6c7c7d4ba0df3b9f696e3de2ebfc0f19a1dcbdf13aea496427 -address /run/k3s/containerd/containerd.sock root 3235 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 587f6221ee9f36c877231ada8d816a799dfda186332c33378d1eb16c72cdc87d -address /run/k3s/containerd/containerd.sock root 4435 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a74c9ec7d99785c2f2d4e6826aa80c22eb8b38249e8f99679ece00a818e9b7b3 -address /run/k3s/containerd/containerd.sock root 4985 1 0 Sep11 ? 00:00:53 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b0c9784dbe0fcbe8be10c857976e70ec84a208cb814e87b5ca085a02d434f8c -address /run/k3s/containerd/containerd.sock root 5056 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id f3231ff35f18056e74eda14907f296be15f7ea1c6ae5ab7904e27d4d18183301 -address /run/k3s/containerd/containerd.sock root 5868 1 0 Sep11 ? 00:00:27 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3e908c4d0b10df275bdef6f72fbcfa09517d11cf749236ad020364e14d77bc93 -address /run/k3s/containerd/containerd.sock root 6158 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id bec5780a5c73fa3154aa3b5ee26cdf23202db821205893e7e66ae17e6103e97b -address /run/k3s/containerd/containerd.sock root 7366 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a81b78845bdcaef710314c93e5ea0d0617f37a8929472f7b570ab90c6667f57f -address /run/k3s/containerd/containerd.sock root 97274 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id c90652c935d9e79af45b9a9ac2b4fe315e2a761a0525737fdb95c680123a164c -address /run/k3s/containerd/containerd.sock root 98309 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 431a2763636488efd7d104c228d41f4f2ecd8c06a7fc375d8977ab0d238936a8 -address /run/k3s/containerd/containerd.sock root 98493 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b480d2fa55c8cc0105ec1902887040c017f03d5b1eb6c73c24bb9d523ad9b37 -address /run/k3s/containerd/containerd.sock +``` + +### 1.2.12 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... +Permissive - Enabling Pod Security Policy can cause applications to unexpectedly fail. + +### 1.2.13 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.14 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.15 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'enable-admission-plugins' +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.16 Ensure that the --secure-port argument is not set to 0 - NoteThis recommendation is obsolete and will be deleted per the consensus process (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'secure-port' +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.17 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'profiling' +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.18 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log +Permissive. + +### 1.2.19 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 +Permissive. + +### 1.2.20 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 +Permissive. + +### 1.2.21 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 +Permissive. + +### 1.2.22 Ensure that the --request-timeout argument is set as appropriate (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, --request-timeout=300s +Permissive. + +### 1.2.23 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.24 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +### 1.2.25 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 1.2.29 +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.26 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep -A1 'Running kube-apiserver' | tail -n2 +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +### 1.2.27 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'client-ca-file' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.28 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-cafile' +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 1.2.29 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= +Permissive - Enabling encryption changes how data can be recovered as data is encrypted. + +### 1.2.30 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. +Permissive - Enabling encryption changes how data can be recovered as data is encrypted. + +### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'tls-cipher-suites' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'terminated-pod-gc-threshold' +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'profiling' +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'use-service-account-credentials' +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'service-account-private-key-file' +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-controller-manager' | tail -n1 | grep 'root-ca-file' +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/k3s/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/k3s/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/k3s/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/k3s/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/k3s/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/k3s/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --terminated-pod-gc-threshold=10 --use-service-account-credentials=true" +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true +Not Applicable. + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep containerd | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 663 1 0 Sep11 ? 00:00:08 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd root 3021 1 0 Sep11 ? 00:00:29 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4790d392966915d995e666002c56ed4cce6dd86f305ce2ee390547a1fcbf6c82 -address /run/k3s/containerd/containerd.sock root 3035 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9d73e6a160ccde6c7c7d4ba0df3b9f696e3de2ebfc0f19a1dcbdf13aea496427 -address /run/k3s/containerd/containerd.sock root 3235 1 0 Sep11 ? 00:00:31 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 587f6221ee9f36c877231ada8d816a799dfda186332c33378d1eb16c72cdc87d -address /run/k3s/containerd/containerd.sock root 4435 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a74c9ec7d99785c2f2d4e6826aa80c22eb8b38249e8f99679ece00a818e9b7b3 -address /run/k3s/containerd/containerd.sock root 4985 1 0 Sep11 ? 00:00:53 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b0c9784dbe0fcbe8be10c857976e70ec84a208cb814e87b5ca085a02d434f8c -address /run/k3s/containerd/containerd.sock root 5056 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id f3231ff35f18056e74eda14907f296be15f7ea1c6ae5ab7904e27d4d18183301 -address /run/k3s/containerd/containerd.sock root 5868 1 0 Sep11 ? 00:00:27 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3e908c4d0b10df275bdef6f72fbcfa09517d11cf749236ad020364e14d77bc93 -address /run/k3s/containerd/containerd.sock root 6158 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id bec5780a5c73fa3154aa3b5ee26cdf23202db821205893e7e66ae17e6103e97b -address /run/k3s/containerd/containerd.sock root 7366 1 0 Sep11 ? 00:00:28 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id a81b78845bdcaef710314c93e5ea0d0617f37a8929472f7b570ab90c6667f57f -address /run/k3s/containerd/containerd.sock root 97274 1 0 16:13 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id c90652c935d9e79af45b9a9ac2b4fe315e2a761a0525737fdb95c680123a164c -address /run/k3s/containerd/containerd.sock root 98309 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 431a2763636488efd7d104c228d41f4f2ecd8c06a7fc375d8977ab0d238936a8 -address /run/k3s/containerd/containerd.sock root 98493 1 0 16:16 ? 00:00:00 /var/lib/rancher/k3s/data/3cdacaf539fc388d8e542a8d643948e3c7bfa4a7e91b7521102325e0ce8581b6/bin/containerd-shim-runc-v2 -namespace k8s.io -id 5b480d2fa55c8cc0105ec1902887040c017f03d5b1eb6c73c24bb9d523ad9b37 -address /run/k3s/containerd/containerd.sock +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-scheduler' | tail -n1 | grep 'bind-address' +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/k3s/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259" +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.1 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/server-client.key +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.2 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.3 +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.4 +``` + +**Expected Result**: + +```console +'cert-file' is present AND 'key-file' is present +``` + +**Returned Value**: + +```console +cert-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.crt key-file: /var/lib/rancher/k3s/server/tls/etcd/peer-server-client.key +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.5 +``` + +**Expected Result**: + +```console +'--client-cert-auth' is present OR 'client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +client-cert-auth: true +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.6 +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +error: process ID list syntax error Usage: ps [options] Try 'ps --help ' or 'ps --help ' for additional help text. For more details see ps(1). cat: /proc//environ: No such file or directory +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/k3s/server/db/etcd/config on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit Script:** `check_for_k3s_etcd.sh` + +```bash +#!/bin/bash + +# This script is used to ensure that k3s is actually running etcd (and not other databases like sqlite3) +# before it checks the requirement +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + + +if [[ "$(journalctl -D /var/log/journal -u k3s | grep 'Managed etcd cluster initializing' | grep -v grep | wc -l)" -gt 0 ]]; then + case $1 in + "1.1.11") + echo $(stat -c %a /var/lib/rancher/k3s/server/db/etcd);; + "1.2.29") + echo $(journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'etcd-');; + "2.1") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.2") + echo $(grep -A 5 'client-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.3") + echo $(grep 'auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.4") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep -E 'cert-file|key-file');; + "2.5") + echo $(grep -A 5 'peer-transport-security' /var/lib/rancher/k3s/server/db/etcd/config | grep 'client-cert-auth');; + "2.6") + echo $(grep 'peer-auto-tls' /var/lib/rancher/k3s/server/db/etcd/config);; + "2.7") + echo $(grep 'trusted-ca-file' /var/lib/rancher/k3s/server/db/etcd/config);; + esac +else +# If another database is running, return whatever is required to pass the scan + case $1 in + "1.1.11") + echo "700";; + "1.2.29") + echo "--etcd-certfile AND --etcd-keyfile";; + "2.1") + echo "cert-file AND key-file";; + "2.2") + echo "--client-cert-auth=true";; + "2.3") + echo "false";; + "2.4") + echo "peer-cert-file AND peer-key-file";; + "2.5") + echo "--client-cert-auth=true";; + "2.6") + echo "--peer-auto-tls=false";; + "2.7") + echo "--trusted-ca-file";; + esac +fi + +``` + +**Audit Execution:** + +```bash +./check_for_k3s_etcd.sh 2.7 +``` + +**Expected Result**: + +```console +'trusted-ca-file' is present +``` + +**Returned Value**: + +```console +trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/server-ca.crt trusted-ca-file: /var/lib/rancher/k3s/server/tls/etcd/peer-ca.crt +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +### 3.1.2 Service account token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of service account tokens. + +### 3.1.3 Bootstrap token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of bootstrap tokens. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Manual) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kube-apiserver' | tail -n1 | grep 'audit-policy-file' +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable - All configuration is passed in as arguments at container run time. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable. + All configuration is passed in as arguments at container run time. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +**Returned Value**: + +```console +600 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/k3s/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/k3s/server/cred/admin.kubeconfig; then stat -c permissions=%a /var/lib/rancher/k3s/server/cred/admin.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/k3s/server/cred/admin.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit:** + +```bash +stat -c %a /var/lib/rancher/k3s/server/tls/server-ca.crt +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +**Returned Value**: + +```console +644 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/k3s/server/tls/client-ca.crt +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 Ensure that the kubelet --config configuration file has permissions set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/kubelet/config.yaml + +### 4.1.10 Ensure that the kubelet --config configuration file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml +Not Applicable. +All configuration is passed in as arguments at container run time. + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "anonymous-auth" | grep -v grep; else echo "--anonymous-auth=false"; fi' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "authorization-mode" | grep -v grep; else echo "--authorization-mode=Webhook"; fi' +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/sh -c 'if test $(journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | wc -l) -gt 0; then journalctl -D /var/log/journal -u k3s | grep "Running kube-apiserver" | tail -n1 | grep "client-ca-file" | grep -v grep; else echo "--client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt"; fi' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +Sep 11 20:52:00 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:00Z" level=info msg="Running kube-apiserver --admission-control-config-file=/etc/rancher/k3s/config/rancher-psact.yaml --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,k3s --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/k3s/server/logs/audit.log --audit-policy-file=/var/lib/rancher/k3s/server/audit.yaml --authorization-mode=Node,RBAC --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/k3s/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/k3s/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/k3s/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/k3s/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/k3s/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/k3s/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/k3s/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/k3s/server/tls/client-auth-proxy.key --request-timeout=300s --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/k3s/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6444 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/k3s/server/tls/service.key --service-account-lookup=true --service-account-signing-key-file=/var/lib/rancher/k3s/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/k3s/server/tls/serving-kube-apiserver.key" +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'read-only-port' +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:15 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:15Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-12-34 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=af02ecbc-1e4e-422e-8b4d-4b2aa24a9d46 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'streaming-connection-idle-timeout' +``` + +### 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +journalctl -D /var/log/journal -u k3s | grep 'Running kubelet' | tail -n1 | grep 'make-iptables-util-chains' +``` + +**Expected Result**: + +```console +'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +Sep 11 20:52:15 ip-172-31-12-34 k3s[2340]: time="2023-09-11T20:52:15Z" level=info msg="Running kubelet --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/k3s/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-12-34 --kubeconfig=/var/lib/rancher/k3s/agent/kubelet.kubeconfig --make-iptables-util-chains=true --node-labels=cattle.io/os=linux,rke.cattle.io/machine=af02ecbc-1e4e-422e-8b4d-4b2aa24a9d46 --pod-infra-container-image=rancher/mirrored-pause:3.6 --pod-manifest-path=/var/lib/rancher/k3s/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/k3s/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/k3s/agent/serving-kubelet.key" +``` + +### 4.2.7 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable. + +### 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--event-qps' is present OR '--event-qps' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** Not Applicable + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service +Permissive - When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. + +### 4.2.10 Ensure that the --rotate-certificates argument is not set to false (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** pass + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable. + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/kubelet/config.yaml +``` + +**Expected Result**: + +```console +'RotateKubeletServerCertificate' is present OR 'RotateKubeletServerCertificate' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +### 4.2.13 Ensure that a limit is set on pod PIDs (Manual) + + +**Result:** warn + +**Remediation:** +Decide on an appropriate level for this parameter and set it, +either via the --pod-max-pids command line parameter or the PodPidsLimit configuration file setting. + +**Audit:** + +```bash +/bin/ps -fC containerd +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--pod-max-pids' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 527 1 0 Sep11 ? 00:01:28 /usr/bin/containerd root 2361 2340 3 Sep11 ? 00:40:19 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[]?.kind=="ServiceAccount" and .subjects[]?.name=="default") or (.subjects[]?.kind=="Group" and .subjects[]?.name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[]? != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +### 5.1.9 Minimize access to create persistent volumes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to PersistentVolume objects in the cluster. + +### 5.1.10 Minimize access to the proxy sub-resource of nodes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the proxy sub-resource of node objects. + +### 5.1.11 Minimize access to the approval sub-resource of certificatesigningrequests objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the approval sub-resource of certificatesigningrequest objects. + +### 5.1.12 Minimize access to webhook configuration objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the validatingwebhookconfigurations or mutatingwebhookconfigurations objects + +### 5.1.13 Minimize access to the service account token creation (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the token sub-resource of serviceaccount objects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +### 5.2.7 Minimize the admission of root containers (Automated) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Automated) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md new file mode 100644 index 00000000000..903e42374f3 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-hardening-guide.md @@ -0,0 +1,515 @@ +--- +title: RKE 加固指南 +--- + + + + + +本文档提供了针对生产环境的 RKE 集群进行加固的具体指导，以便在使用 Rancher 部署之前进行配置。它概述了满足信息安全中心（Center for Information Security, CIS）Kubernetes benchmark controls 所需的配置和控制。 + +:::note +这份加固指南描述了如何确保你集群中的节点安全。我们建议你在安装 Kubernetes 之前遵循本指南。 +::: + +此加固指南适用于 RKE 集群，并与以下版本的 CIS Kubernetes Benchmark、Kubernetes 和 Rancher 相关联： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|------------------------------| +| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 至 v1.26 | + +:::note +- 在 Benchmark v1.24 及更高版本中，检查 id `4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automated)` 可能会失败，因为 `/etc/kubernetes/ssl/kube-ca.pem` 默认设置为 644。 +- 在 Benchmark v1.7 中，不再需要 `--protect-kernel-defaults` (`4.2.6`) 参数，并已被 CIS 删除。 +::: + +有关如何评估加固的 RKE 集群与官方 CIS benchmark 的更多细节，请参考特定 Kubernetes 和 CIS benchmark 版本的 RKE 自我评估指南。 + +## 主机级别要求 + +### 配置 Kernel 运行时参数 + +建议对群集中的所有节点类型使用以下 `sysctl` 配置。在 `/etc/sysctl.d/90-kubelet.conf` 中设置以下参数： + +```ini +vm.overcommit_memory=1 +vm.panic_on_oom=0 +kernel.panic=10 +kernel.panic_on_oops=1 +``` + +运行 `sysctl -p /etc/sysctl.d/90-kubelet.conf` 以启用设置。 + +### 配置 `etcd` 用户和组 + +在安装 RKE 之前，需要设置 **etcd** 服务的用户帐户和组。 + +#### 创建 `etcd` 用户和组 + +要创建 **etcd** 用户和组，请运行以下控制台命令。 +下面的命令示例中使用 `52034` 作为 **uid** 和 **gid** 。 +任何有效且未使用的 **uid** 或 **gid** 都可以代替 `52034`。 + +```bash +groupadd --gid 52034 etcd +useradd --comment "etcd service account" --uid 52034 --gid 52034 etcd --shell /usr/sbin/nologin +``` + +在通过集群配置文件 `config.yml` 部署RKE时，请更新 `etcd` 用户的 `uid` 和 `gid`： + +```yaml +services: + etcd: + gid: 52034 + uid: 52034 +``` + +## Kubernetes 运行时要求 + +### 配置 `default` Service Account + +#### 设置 `automountServiceAccountToken` 为 `false` 用于 `default` service accounts + +Kubernetes 提供了一个 default service account，供集群工作负载使用，其中没有为 pod 分配特定的 service account。 +如果需要从 pod 访问 Kubernetes API，则应为该 pod 创建特定的 service account，并向该 service account 授予权限。 +应配置 default service account，使其不提供 service account 令牌，并且不应具有任何明确的权限分配。 + +对于标准 RKE 安装上的每个命名空间（包括 `default` 和 `kube-system`），`default` service account 必须包含以下值： + +```yaml +automountServiceAccountToken: false +``` + +将以下配置保存到名为 `account_update.yaml` 的文件中。 + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: default +automountServiceAccountToken: false +``` + +创建一个名为 `account_update.yaml` 的 bash 脚本文件。 +确保执行 `chmod +x account_update.sh` 命令，以赋予脚本执行权限。 + +```bash +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" +done +``` + +执行此脚本将 `account_update.yaml` 配置应用到所有命名空间中的 `default` service account。 + +### 配置网络策略 + +#### 确保所有命名空间都定义了网络策略 + +在同一个 Kubernetes 集群上运行不同的应用程序会带来风险，即某个受感染的应用程序可能会攻击相邻的应用程序。为确保容器只与其预期通信的容器进行通信，网络分段至关重要。网络策略规定了哪些 Pod 可以互相通信，以及与其他网络终端通信的方式。 + +网络策略是命名空间范围的。当在特定命名空间引入网络策略时，所有未被策略允许的流量将被拒绝。然而，如果在命名空间中没有网络策略，那么所有流量将被允许进入和离开该命名空间中的 Pod。要强制执行网络策略，必须启用容器网络接口（container network interface, CNI）插件。本指南使用 [Canal](https://github.com/projectcalico/canal) 来提供策略执行。有关 CNI 提供程序的其他信息可以在[这里](https://www.suse.com/c/rancher_blog/comparing-kubernetes-cni-providers-flannel-calico-canal-and-weave/)找到。 + +一旦在集群上启用了 CNI 提供程序，就可以应用默认的网络策略。下面提供了一个 **permissive** 的示例供参考。如果你希望允许匹配某个命名空间中所有 Pod 的所有入站和出站流量（即使添加了策略导致某些 Pod 被视为”隔离”），你可以创建一个明确允许该命名空间中所有流量的策略。请将以下配置保存为 `default-allow-all.yaml`。有关网络策略的其他[文档](https://kubernetes.io/docs/concepts/services-networking/network-policies/)可以在 Kubernetes 站点上找到。 + +:::caution +此网络策略只是一个示例，不建议用于生产用途。 +::: + +```yaml +--- +apiVersion: networking.k8s.io/v1 +kind: NetworkPolicy +metadata: + name: default-allow-all +spec: + podSelector: {} + ingress: + - {} + egress: + - {} + policyTypes: + - Ingress + - Egress +``` + +创建一个名为 `apply_networkPolicy_to_all_ns.sh`的 Bash 脚本文件。 + +确保运行 `chmod +x apply_networkPolicy_to_all_ns.sh` 命令，以赋予脚本执行权限。 + +```bash +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + kubectl apply -f default-allow-all.yaml -n ${namespace} +done +``` + +执行此脚本以将 `default-allow-all.yaml` 配置和 **permissive** 的 `NetworkPolicy` 应用于所有命名空间。 + +## 已知限制 + +- 当注册自定义节点仅提供公共 IP 时，Rancher **exec shell** 和 **查看 pod 日志** 在加固设置中**不起作用**。 此功能需要在注册自定义节点时提供私有 IP。 +- 当根据 Rancher [提供](../../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/create-pod-security-policies.md)的 Pod 安全策略 (Pod Security Policies, PSP) 将 `default_pod_security_policy_template_id:` 设置为 `restricted` 或 `restricted-noroot` 时，Rancher 会在 `default` service accounts 上创建 `RoleBindings` 和 `ClusterRoleBindings`。CIS 检查 5.1.5 要求除了默认角色之外，`default` service accounts 不应绑定其他角色或集群角色。此外，`default` service accounts 应配置为不提供服务账户令牌，也不具有任何明确的权限分配。 + +## 加固的 RKE `cluster.yml` 配置参考 + +参考的 `cluster.yml` 文件是由 RKE CLI 使用的，它提供了实现 RKE 加固安装所需的配置。 +RKE [文档](https://rancher.com/docs/rke/latest/en/installation/)提供了有关配置项的更多详细信息。这里参考的 `cluster.yml` 不包括必需的 `nodes` 指令，因为它取决于你的环境。在 RKE 中有关节点配置的文档可以在[这里](https://rancher.com/docs/rke/latest/en/config-options/nodes/)找到。 + +示例 `cluster.yml` 配置文件中包含了一个 Admission Configuration 策略，在 `services.kube-api.admission_configuration` 字段中指定。这个[示例](../../psa-restricted-exemptions.md)策略包含了命名空间的豁免规则，这对于在Rancher中正确运行导入的RKE集群非常必要，类似于Rancher预定义的 [`rancher-restricted`](../../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md) 策略。 + +如果你希望使用 RKE 的默认 `restricted` 策略，则将 `services.kube-api.admission_configuration` 字段留空，并将 `services.pod_security_configuration` 设置为 `restricted`。你可以在 [RKE 文档](https://rke.docs.rancher.com/config-options/services/pod-security-admission)中找到更多信息。 + + + + +:::note +如果你打算将一个 RKE 集群导入到 Rancher 中，请参考此[文档](../../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md)以了解如何配置 PSA 以豁免 Rancher 系统命名空间。 +::: + +```yaml +# 如果你打算在离线环境部署 Kubernetes， +# 请查阅文档以了解如何配置自定义的 RKE 镜像。 +nodes: [] +kubernetes_version: # 定义 RKE 版本 +services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + secrets_encryption_config: + enabled: true + audit_log: + enabled: true + event_rate_limit: + enabled: true + # 如果你在 `admission_configuration` 中设置了自定义策略， + # 请将 `pod_security_configuration` 字段留空。 + # 否则，将其设置为 `restricted` 以使用 RKE 预定义的受限策略， + # 并删除 `admission_configuration` 字段中的所有内容。 + # + # pod_security_configuration: restricted + # + admission_configuration: + apiVersion: apiserver.config.k8s.io/v1 + kind: AdmissionConfiguration + plugins: + - name: PodSecurity + configuration: + apiVersion: pod-security.admission.config.k8s.io/v1 + kind: PodSecurityConfiguration + defaults: + enforce: "restricted" + enforce-version: "latest" + audit: "restricted" + audit-version: "latest" + warn: "restricted" + warn-version: "latest" + exemptions: + usernames: [] + runtimeClasses: [] + namespaces: [calico-apiserver, + calico-system, + cattle-alerting, + cattle-csp-adapter-system, + cattle-elemental-system, + cattle-epinio-system, + cattle-externalip-system, + cattle-fleet-local-system, + cattle-fleet-system, + cattle-gatekeeper-system, + cattle-global-data, + cattle-global-nt, + cattle-impersonation-system, + cattle-istio, + cattle-istio-system, + cattle-logging, + cattle-logging-system, + cattle-monitoring-system, + cattle-neuvector-system, + cattle-prometheus, + cattle-provisioning-capi-system, + cattle-resources-system, + cattle-sriov-system, + cattle-system, + cattle-ui-plugin-system, + cattle-windows-gmsa-system, + cert-manager, + cis-operator-system, + fleet-default, + ingress-nginx, + istio-system, + kube-node-lease, + kube-public, + kube-system, + longhorn-system, + rancher-alerting-drivers, + security-scan, + tigera-operator] + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + generate_serving_certificate: true +addons: | + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: default-allow-all + spec: + podSelector: {} + ingress: + - {} + egress: + - {} + policyTypes: + - Ingress + - Egress + --- + apiVersion: v1 + kind: ServiceAccount + metadata: + name: default + automountServiceAccountToken: false +``` + + + + +```yaml +# 如果你打算在离线环境部署 Kubernetes， +# 请查阅文档以了解如何配置自定义的 RKE 镜像。 +nodes: [] +kubernetes_version: # 定义 RKE 版本 +services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + secrets_encryption_config: + enabled: true + audit_log: + enabled: true + event_rate_limit: + enabled: true + pod_security_policy: true + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: true + generate_serving_certificate: true +addons: | + # Upstream Kubernetes restricted PSP policy + # https://github.com/kubernetes/website/blob/564baf15c102412522e9c8fc6ef2b5ff5b6e766c/content/en/examples/policy/restricted-psp.yaml + apiVersion: policy/v1beta1 + kind: PodSecurityPolicy + metadata: + name: restricted-noroot + spec: + privileged: false + # Required to prevent escalations to root. + allowPrivilegeEscalation: false + requiredDropCapabilities: + - ALL + # Allow core volume types. + volumes: + - 'configMap' + - 'emptyDir' + - 'projected' + - 'secret' + - 'downwardAPI' + # Assume that ephemeral CSI drivers & persistentVolumes set up by the cluster admin are safe to use. + - 'csi' + - 'persistentVolumeClaim' + - 'ephemeral' + hostNetwork: false + hostIPC: false + hostPID: false + runAsUser: + # Require the container to run without root privileges. + rule: 'MustRunAsNonRoot' + seLinux: + # This policy assumes the nodes are using AppArmor rather than SELinux. + rule: 'RunAsAny' + supplementalGroups: + rule: 'MustRunAs' + ranges: + # Forbid adding the root group. + - min: 1 + max: 65535 + fsGroup: + rule: 'MustRunAs' + ranges: + # Forbid adding the root group. + - min: 1 + max: 65535 + readOnlyRootFilesystem: false + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRole + metadata: + name: psp:restricted-noroot + rules: + - apiGroups: + - extensions + resourceNames: + - restricted-noroot + resources: + - podsecuritypolicies + verbs: + - use + --- + apiVersion: rbac.authorization.k8s.io/v1 + kind: ClusterRoleBinding + metadata: + name: psp:restricted-noroot + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: psp:restricted-noroot + subjects: + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:serviceaccounts + - apiGroup: rbac.authorization.k8s.io + kind: Group + name: system:authenticated + --- + apiVersion: networking.k8s.io/v1 + kind: NetworkPolicy + metadata: + name: default-allow-all + spec: + podSelector: {} + ingress: + - {} + egress: + - {} + policyTypes: + - Ingress + - Egress + --- + apiVersion: v1 + kind: ServiceAccount + metadata: + name: default + automountServiceAccountToken: false +``` + + + + +## 加固后的 RKE 集群模板配置参考 + +参考的 RKE 集群模板提供了实现 Kubernetes 加固安装所需的最低配置。RKE 模板用于提供 Kubernetes 并定义 Rancher 设置。有关安装 RKE 及其模板详情的其他信息，请参考 Rancher [文档](../../../../getting-started/installation-and-upgrade/installation-and-upgrade.md) 。 + + + + +```yaml +# +# 集群配置 +# +default_pod_security_admission_configuration_template_name: rancher-restricted +enable_network_policy: true +local_cluster_auth_endpoint: + enabled: true +name: # 定义集群名称 + +# +# Rancher 配置 +# +rancher_kubernetes_engine_config: + addon_job_timeout: 45 + authentication: + strategy: x509|webhook + kubernetes_version: # 定义 RKE 版本 + services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + audit_log: + enabled: true + event_rate_limit: + enabled: true + pod_security_policy: false + secrets_encryption_config: + enabled: true + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + generate_serving_certificate: true + scheduler: + extra_args: + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + + + + +```yaml +# +# 集群配置 +# +default_pod_security_policy_template_id: restricted-noroot +enable_network_policy: true +local_cluster_auth_endpoint: + enabled: true +name: # 定义集群名称 + +# +# Rancher 配置 +# +rancher_kubernetes_engine_config: + addon_job_timeout: 45 + authentication: + strategy: x509|webhook + kubernetes_version: # 定义 RKE 版本 + services: + etcd: + uid: 52034 + gid: 52034 + kube-api: + audit_log: + enabled: true + event_rate_limit: + enabled: true + pod_security_policy: true + secrets_encryption_config: + enabled: true + kube-controller: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + kubelet: + extra_args: + feature-gates: RotateKubeletServerCertificate=true + protect-kernel-defaults: true + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 + generate_serving_certificate: true + scheduler: + extra_args: + tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + + + + +## 结论 + +如果你按照本指南操作，由 Rancher 提供的 RKE 自定义集群将配置为通过 CIS Kubernetes Benchmark 测试。你可以查看我们的 RKE 自我评估指南，了解我们是如何验证每个 benchmarks 的，并且你可以在你的集群上执行相同的操作。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md index 34bb63f89f0..7dfb1204916 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md @@ -1,30 +1,34 @@ --- -title: RKE Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.23 +title: RKE 自我评估指南 - CIS Benchmark v1.23 - K8s v1.23 --- -This document is a companion to the [RKE Hardening Guide](../../../../pages-for-subheaders/rke1-hardening-guide.md), which provides prescriptive guidance on how to harden RKE clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. + + + -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: +本文档是 [RKE 加固指南](rke1-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 -| Rancher Version | CIS Benchmark Version | Kubernetes Version | +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | |-----------------|-----------------------|--------------------| | Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 -This document is for Rancher operators, security teams, auditors and decision makers. +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.23 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 -## Testing Methodology +## 测试方法 -Rancher and RKE install Kubernetes services via Docker containers. Configuration is defined by arguments passed to the container at the time of initialization, not via configuration files. +Rancher 和 RKE 通过 Docker 容器安装 Kubernetes 服务。配置是通过初始化时传递给容器的参数定义的，而不是通过配置文件。 -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 :::note -This guide only covers `automated` (previously called `scored`) tests. +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md new file mode 100644 index 00000000000..2246a13f6d0 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md @@ -0,0 +1,3046 @@ +--- +title: RKE 自我评估指南 - CIS Benchmark v1.24 - K8s v1.24 +--- + + + + + +本文档是 [RKE 加固指南](rke1-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | + +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.24 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +Rancher 和 RKE 通过 Docker 容器安装 Kubernetes 服务。配置是通过初始化时传递给容器的参数定义的，而不是通过配置文件。 + +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. +All configuration is passed in as arguments at container run time. + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for etcd. +All configuration is passed in as arguments at container run time. + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=644 +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root root:root +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c %a /node/var/lib/etcd +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** Not Applicable + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the master node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit Script:** `check_files_owner_in_dir.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to ensure the owner is set to root:root for +# the given directory and all the files in it +# +# inputs: +# $1 = /full/path/to/directory +# +# outputs: +# true/false + +INPUT_DIR=$1 + +if [[ "${INPUT_DIR}" == "" ]]; then + echo "false" + exit +fi + +if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then + echo "false" + exit +fi + +statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) +while read -r statInfoLine; do + f=$(echo ${statInfoLine} | cut -d' ' -f1) + p=$(echo ${statInfoLine} | cut -d' ' -f2) + + if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then + if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "root:root" ]]; then + echo "false" + exit + fi + fi +done <<< "${statInfoLines}" + + +echo "true" +exit + +``` + +**Audit Execution:** + +```bash +./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' -exec chmod -R 600 {} + + +**Audit:** + +```bash +find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' | xargs stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=644 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /etc/kubernetes/ssl/*key.pem + +**Audit Script:** `check_files_permissions.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to ensure the file permissions are set to 644 or +# more restrictive for all files in a given directory or a wildcard +# selection of files +# +# inputs: +# $1 = /full/path/to/directory or /path/to/fileswithpattern +# ex: !(*key).pem +# +# $2 (optional) = permission (ex: 600) +# +# outputs: +# true/false + +# Turn on "extended glob" for use of '!' in wildcard +shopt -s extglob + +# Turn off history to avoid surprises when using '!' +set -H + +USER_INPUT=$1 + +if [[ "${USER_INPUT}" == "" ]]; then + echo "false" + exit +fi + + +if [[ -d ${USER_INPUT} ]]; then + PATTERN="${USER_INPUT}/*" +else + PATTERN="${USER_INPUT}" +fi + +PERMISSION="" +if [[ "$2" != "" ]]; then + PERMISSION=$2 +fi + +FILES_PERMISSIONS=$(stat -c %n\ %a ${PATTERN}) + +while read -r fileInfo; do + p=$(echo ${fileInfo} | cut -d' ' -f2) + + if [[ "${PERMISSION}" != "" ]]; then + if [[ "$p" != "${PERMISSION}" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "644" && "$p" != "640" && "$p" != "600" ]]; then + echo "false" + exit + fi + fi +done <<< "${FILES_PERMISSIONS}" + + +echo "true" +exit + +``` + +**Audit Execution:** + +```bash +./check_files_permissions.sh '/node/etc/kubernetes/ssl/*key.pem' +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --kubelet-https parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-https' is present OR '--kubelet-https' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.18 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxsize' is greater or equal to 100 +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 1.2.31 Ensure that encryption providers are appropriately configured (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit Script:** `check_encryption_provider_config.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to check the encrption provider config is set to aesbc +# +# outputs: +# true/false + +# TODO: Figure out the file location from the kube-apiserver commandline args +ENCRYPTION_CONFIG_FILE="/node/etc/kubernetes/ssl/encryption.yaml" + +if [[ ! -f "${ENCRYPTION_CONFIG_FILE}" ]]; then + echo "false" + exit +fi + +for provider in "$@" +do + if grep "$provider" "${ENCRYPTION_CONFIG_FILE}"; then + echo "true" + exit + fi +done + +echo "false" +exit + +``` + +**Audit Execution:** + +```bash +./check_encryption_provider_config.sh aescbc +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +- aescbc: true +``` + +### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +Cluster provisioned by RKE handles certificate rotation directly through RKE. + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 3690 3671 1 Sep11 ? 00:20:42 kube-controller-manager --service-cluster-ip-range=10.43.0.0/16 --configure-cloud-routes=false --enable-hostpath-provisioner=false --terminated-pod-gc-threshold=1000 --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --v=2 --pod-eviction-timeout=5m0s --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --cloud-provider= --cluster-cidr=10.42.0.0/16 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --allocate-node-cidrs=true --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --feature-gates=RotateKubeletServerCertificate=true --leader-elect=true --profiling=false --node-monitor-grace-period=40s --allow-untagged-cloud=true --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 3859 3838 0 Sep11 ? 00:03:44 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 3859 3838 0 Sep11 ? 00:03:44 kube-scheduler --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --leader-elect=true --profiling=false --v=2 --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--cert-file' is present AND '--key-file' is present +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:08 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 5 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:08 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 4 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-7-100 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-cert-file' is present AND '--peer-key-file' is present +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 2 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 2 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-7-100 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--trusted-ca-file' is present +``` + +**Returned Value**: + +```console +etcd 3369 3348 2 Sep11 ? 00:26:05 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-cluster-token=etcd-cluster-1 --initial-cluster=etcd-ip-172-31-7-100=https://172.31.7.100:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --client-cert-auth=true --election-timeout=5000 --name=etcd-ip-172-31-7-100 --listen-client-urls=https://0.0.0.0:2379 --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-state=new --advertise-client-urls=https://172.31.7.100:2379 --heartbeat-interval=500 --initial-advertise-peer-urls=https://172.31.7.100:2380 --listen-peer-urls=https://0.0.0.0:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-7-100-key.pem --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 root 3528 3509 7 Sep11 ? 01:24:09 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s root 1057543 1057522 2 16:15 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.24-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +root 3528 3509 7 Sep11 ? 01:24:10 kube-apiserver --service-cluster-ip-range=10.43.0.0/16 --anonymous-auth=false --profiling=false --advertise-address=172.31.7.100 --audit-log-maxsize=100 --cloud-provider= --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --etcd-prefix=/registry --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --requestheader-username-headers=X-Remote-User --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --audit-policy-file=/etc/kubernetes/audit-policy.yaml --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --storage-backend=etcd3 --service-node-port-range=30000-32767 --bind-address=0.0.0.0 --api-audiences=unknown --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --runtime-config=policy/v1beta1/podsecuritypolicy=true --allow-privileged=true --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,PodSecurityPolicy,EventRateLimit --service-account-issuer=rke --requestheader-allowed-names=kube-apiserver-proxy-client --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-format=json --authorization-mode=Node,RBAC --etcd-servers=https://172.31.7.100:2379 --service-account-lookup=true --secure-port=6443 --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --audit-log-maxage=30 --admission-control-config-file=/etc/kubernetes/admission.yaml --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --audit-log-maxbackup=10 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --authentication-token-webhook-cache-ttl=5s +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. +All configuration is passed in as arguments at container run time. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Cluster provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. +All configuration is passed in as arguments at container run time. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit:** + +```bash +stat -c permissions=%a /node/etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit:** + +```bash +stat -c %U:%G /node/etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/kubelet/config.yaml + +Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml + +Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--streaming-connection-idle-timeout' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--protect-kernel-defaults' is equal to 'true' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors + +### 4.2.9 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--event-qps' is equal to '0' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +Clusters provisioned by RKE handles certificate rotation directly through RKE. + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4429 4031 3 Sep11 ? 00:38:25 kubelet --read-only-port=0 --event-qps=0 --root-dir=/var/lib/kubelet --pod-infra-container-image=rancher/mirrored-pause:3.7 --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100-key.pem --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --cloud-provider= --address=0.0.0.0 --protect-kernel-defaults=true --hostname-override=ip-172-31-7-100 --resolv-conf=/etc/resolv.conf --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-7-100.pem --make-iptables-util-chains=true --feature-gates=RotateKubeletServerCertificate=true --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --container-runtime=remote --v=2 --anonymous-auth=false --authentication-token-webhook=true --fail-swap-on=false --cgroups-per-qos=True --authorization-mode=Webhook --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Automated) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[]?.kind=="ServiceAccount" and .subjects[]?.name=="default") or (.subjects[]?.kind=="Group" and .subjects[]?.name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[]? != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "ingress-nginx" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" Error from server (Forbidden): roles.rbac.authorization.k8s.io "default-psp-role" is forbidden: User "system:serviceaccount:cis-operator-system:cis-serviceaccount" cannot get resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "kube-system" true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=1 +``` + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=1 +``` + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=1 +``` + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=1 +``` + +### 5.2.7 Minimize the admission of root containers (Manual) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +**Audit Script:** `check_for_network_policies.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +for namespace in $(kubectl get namespaces --all-namespaces -o json | jq -r '.items[].metadata.name'); do + policy_count=$(kubectl get networkpolicy -n ${namespace} -o json | jq '.items | length') + if [[ ${policy_count} -eq 0 ]]; then + echo "false" + exit + fi +done + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_network_policies.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Automated) + + +**Result:** pass + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + +**Audit Script:** `check_for_default_ns.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count=$(kubectl get all -n default -o json | jq .items[] | jq -r 'select((.metadata.name!="kubernetes"))' | jq .metadata.name | wc -l) +if [[ ${count} -gt 0 ]]; then + echo "false" + exit +fi + +echo "true" + + +``` + +**Audit Execution:** + +```bash +./check_for_default_ns.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md new file mode 100644 index 00000000000..5e5a6b1eed2 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke1-hardening-guide/rke1-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md @@ -0,0 +1,2862 @@ +--- +title: RKE 自我评估指南 - CIS Benchmark v1.7 - K8s v1.25/v1.26/v1.27 +--- + + + + + +本文档是 [RKE 加固指南](rke1-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25/v1.26/v1.27 | + +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.7 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +Rancher 和 RKE 通过 Docker 容器安装 Kubernetes 服务。配置是通过初始化时传递给容器的参数定义的，而不是通过配置文件。 + +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-apiserver.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-apiserver.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-controller-manager.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-controller-manager.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/manifests/kube-scheduler.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/manifests/kube-scheduler.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /etc/kubernetes/manifests/etcd.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /etc/kubernetes/manifests/etcd.yaml +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for kube-apiserver. +All configuration is passed in as arguments at container run time. + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +**Audit:** + +```bash +ps -ef | grep kubelet | grep -- --cni-conf-dir | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a +``` + +**Expected Result**: + +```console +'permissions' is present +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +**Audit:** + +```bash +ps -ef | grep kubelet | grep -- --cni-conf-dir | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c %a /node/var/lib/etcd +``` + +**Expected Result**: + +```console +'700' is equal to '700' +``` + +**Returned Value**: + +```console +700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +**Audit:** + +```bash +stat -c %U:%G /node/var/lib/etcd +``` + +**Expected Result**: + +```console +'etcd:etcd' is present +``` + +**Returned Value**: + +```console +etcd:etcd +``` + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/admin.conf +Not Applicable - Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf +Not Applicable - Cluster provisioned by RKE does not store the kubernetes default kubeconfig credentials file on the nodes. + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 scheduler +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root scheduler +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for scheduler. +All configuration is passed in as arguments at container run time. + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 controllermanager +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root controllermanager +Not Applicable - Cluster provisioned by RKE doesn't require or maintain a configuration file for controller-manager. +All configuration is passed in as arguments at container run time. + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit Script:** `check_files_owner_in_dir.sh` + +```bash +#!/usr/bin/env bash + +# This script is used to ensure the owner is set to root:root for +# the given directory and all the files in it +# +# inputs: +# $1 = /full/path/to/directory +# +# outputs: +# true/false + +INPUT_DIR=$1 + +if [[ "${INPUT_DIR}" == "" ]]; then + echo "false" + exit +fi + +if [[ $(stat -c %U:%G ${INPUT_DIR}) != "root:root" ]]; then + echo "false" + exit +fi + +statInfoLines=$(stat -c "%n %U:%G" ${INPUT_DIR}/*) +while read -r statInfoLine; do + f=$(echo ${statInfoLine} | cut -d' ' -f1) + p=$(echo ${statInfoLine} | cut -d' ' -f2) + + if [[ $(basename "$f" .pem) == "kube-etcd-"* ]]; then + if [[ "$p" != "root:root" && "$p" != "etcd:etcd" ]]; then + echo "false" + exit + fi + else + if [[ "$p" != "root:root" ]]; then + echo "false" + exit + fi + fi +done <<< "${statInfoLines}" + + +echo "true" +exit + +``` + +**Audit Execution:** + +```bash +./check_files_owner_in_dir.sh /node/etc/kubernetes/ssl +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' -exec chmod -R 600 {} + + +**Audit:** + +```bash +find /node/etc/kubernetes/ssl/ -name '*.pem' ! -name '*key.pem' | xargs stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=644 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +find /node/etc/kubernetes/ssl/ -name '*key.pem' -exec chmod -R 600 {} + + +**Audit:** + +```bash +find /node/etc/kubernetes/ssl/ -name '*key.pem' | xargs stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/etc/kubernetes/manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= +When generating serving certificates, functionality could break in conjunction with hostname overrides which are required for certain cloud providers. + +### 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.7 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.9 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.12 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.13 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.14 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.15 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.16 Ensure that the --secure-port argument is not set to 0 - NoteThis recommendation is obsolete and will be deleted per the consensus process (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.17 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.18 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.19 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.20 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.21 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxsize' is greater or equal to 100 +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.22 Ensure that the --request-timeout argument is set as appropriate (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, --request-timeout=300s + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +### 1.2.23 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is equal to 'true' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.24 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.25 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.26 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.27 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.28 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.29 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 1.2.30 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit:** + +```bash +ENCRYPTION_PROVIDER_CONFIG=$(ps -ef | grep kube-apiserver | grep -- --encryption-provider-config | sed 's%.*encryption-provider-config[= ]\([^ ]*\).*%\1%') if test -e $ENCRYPTION_PROVIDER_CONFIG; then grep -A1 'providers:' $ENCRYPTION_PROVIDER_CONFIG | tail -n1 | grep -o "[A-Za-z]*" | sed 's/^/provider=/'; fi +``` + +**Expected Result**: + +```console +'provider' is present +``` + +### 1.2.31 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384' +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true +Cluster provisioned by RKE handles certificate rotation directly through RKE. + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /etc/kubernetes/manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 4184 4163 1 Sep11 ? 00:20:06 kube-controller-manager --configure-cloud-routes=false --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --allocate-node-cidrs=true --enable-hostpath-provisioner=false --pod-eviction-timeout=5m0s --terminated-pod-gc-threshold=1000 --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --service-cluster-ip-range=10.43.0.0/16 --cluster-cidr=10.42.0.0/16 --root-ca-file=/etc/kubernetes/ssl/kube-ca.pem --service-account-private-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --node-monitor-grace-period=40s --v=2 --profiling=false --cloud-provider= --allow-untagged-cloud=true --leader-elect=true --feature-gates=RotateKubeletServerCertificate=true --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-controller-manager.yaml --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 4339 4318 0 Sep11 ? 00:03:28 kube-scheduler --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --profiling=false --v=2 --leader-elect=true +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /etc/kubernetes/manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is present OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 4339 4318 0 Sep11 ? 00:03:28 kube-scheduler --authentication-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --authorization-kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-scheduler.yaml --profiling=false --v=2 --leader-elect=true +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--cert-file' is present AND '--key-file' is present +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 2 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 1 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-4-224 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-cert-file' is present AND '--peer-key-file' is present +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 2 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--peer-client-cert-auth' is equal to 'true' +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 1 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-4-224 ETCDCTL_API=3 ETCDCTL_CACERT=/etc/kubernetes/ssl/kube-ca.pem ETCDCTL_CERT=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem ETCDCTL_KEY=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem ETCDCTL_ENDPOINTS=https://127.0.0.1:2379 ETCD_UNSUPPORTED_ARCH=x86_64 HOME=/ +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'--trusted-ca-file' is present +``` + +**Returned Value**: + +```console +etcd 3847 3824 2 Sep11 ? 00:29:36 /usr/local/bin/etcd --peer-client-cert-auth=true --initial-advertise-peer-urls=https://172.31.4.224:2380 --initial-cluster=etcd-ip-172-31-4-224=https://172.31.4.224:2380 --initial-cluster-state=new --trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --heartbeat-interval=500 --listen-client-urls=https://0.0.0.0:2379 --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --listen-peer-urls=https://0.0.0.0:2380 --cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --peer-cert-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224.pem --peer-key-file=/etc/kubernetes/ssl/kube-etcd-172-31-4-224-key.pem --data-dir=/var/lib/rancher/etcd/ --initial-cluster-token=etcd-cluster-1 --name=etcd-ip-172-31-4-224 --advertise-client-urls=https://172.31.4.224:2379 --cipher-suites=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 --election-timeout=5000 root 4018 3998 5 Sep11 ? 01:03:21 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml root 1034677 1034607 1 16:16 ? 00:00:00 kube-bench run --targets etcd --scored --nosummary --noremediations --v=0 --config-dir=/etc/kube-bench/cfg --benchmark rke-cis-1.7-hardened --json --log_dir /tmp/sonobuoy/logs --outputfile /tmp/sonobuoy/etcd.json +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +### 3.1.2 Service account token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of service account tokens. + +### 3.1.3 Bootstrap token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of bootstrap tokens. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +root 4018 3998 5 Sep11 ? 01:03:22 kube-apiserver --advertise-address=172.31.4.224 --audit-log-path=/var/log/kube-audit/audit-log.json --audit-log-maxbackup=10 --requestheader-allowed-names=kube-apiserver-proxy-client --service-cluster-ip-range=10.43.0.0/16 --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota,NodeRestriction,Priority,TaintNodesByCondition,PersistentVolumeClaimResize,EventRateLimit --requestheader-extra-headers-prefix=X-Remote-Extra- --tls-private-key-file=/etc/kubernetes/ssl/kube-apiserver-key.pem --storage-backend=etcd3 --anonymous-auth=false --bind-address=0.0.0.0 --cloud-provider= --etcd-certfile=/etc/kubernetes/ssl/kube-node.pem --requestheader-client-ca-file=/etc/kubernetes/ssl/kube-apiserver-requestheader-ca.pem --service-node-port-range=30000-32767 --profiling=false --proxy-client-key-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client-key.pem --encryption-provider-config=/etc/kubernetes/ssl/encryption.yaml --runtime-config=authorization.k8s.io/v1beta1=true --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --service-account-lookup=true --etcd-servers=https://172.31.4.224:2379 --api-audiences=unknown --requestheader-group-headers=X-Remote-Group --service-account-issuer=rke --audit-log-maxsize=100 --service-account-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --secure-port=6443 --service-account-signing-key-file=/etc/kubernetes/ssl/kube-service-account-token-key.pem --authorization-mode=Node,RBAC --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --audit-log-maxage=30 --audit-log-format=json --etcd-prefix=/registry --kubelet-client-certificate=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-config-file=/etc/kubernetes/kube-api-authn-webhook.yaml --kubelet-certificate-authority=/etc/kubernetes/ssl/kube-ca.pem --kubelet-client-key=/etc/kubernetes/ssl/kube-apiserver-key.pem --proxy-client-cert-file=/etc/kubernetes/ssl/kube-apiserver-proxy-client.pem --tls-cert-file=/etc/kubernetes/ssl/kube-apiserver.pem --authentication-token-webhook-cache-ttl=5s --admission-control-config-file=/etc/kubernetes/admission.yaml --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --etcd-cafile=/etc/kubernetes/ssl/kube-ca.pem --etcd-keyfile=/etc/kubernetes/ssl/kube-node-key.pem --requestheader-username-headers=X-Remote-User --allow-privileged=true --audit-policy-file=/etc/kubernetes/audit-policy.yaml +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable - Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. +All configuration is passed in as arguments at container run time. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable - Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet service. + All configuration is passed in as arguments at container run time. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-proxy.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c permissions=%a /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; then stat -c %U:%G /node/etc/kubernetes/ssl/kubecfg-kube-node.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit:** + +```bash +stat -c permissions=%a /node/etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit:** + +```bash +stat -c %U:%G /node/etc/kubernetes/ssl/kube-ca.pem +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/kubelet/config.yaml +Not Applicable - Clusters provisioned by RKE do not require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/kubelet/config.yaml +Not Applicable - Clusters provisioned by RKE doesn’t require or maintain a configuration file for the kubelet. +All configuration is passed in as arguments at container run time. + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--streaming-connection-idle-timeout' is not equal to '0' OR '--streaming-connection-idle-timeout' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--make-iptables-util-chains' is equal to 'true' OR '--make-iptables-util-chains' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.7 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable - Clusters provisioned by RKE set the --hostname-override to avoid any hostname configuration errors + +### 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--event-qps' is greater or equal to 0 OR '--event-qps' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.10 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--rotate-certificates' is present OR '--rotate-certificates' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable - Clusters provisioned by RKE handles certificate rotation directly through RKE. + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +### 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--tls-cipher-suites' contains valid elements from 'TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +### 4.2.13 Ensure that a limit is set on pod PIDs (Manual) + + +**Result:** warn + +**Remediation:** +Decide on an appropriate level for this parameter and set it, +either via the --pod-max-pids command line parameter or the PodPidsLimit configuration file setting. + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/sh -c 'if test -e /var/lib/kubelet/config.yaml; then /bin/cat /var/lib/kubelet/config.yaml; fi' +``` + +**Expected Result**: + +```console +'--pod-max-pids' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 4903 4499 3 Sep11 ? 00:36:52 kubelet --v=2 --client-ca-file=/etc/kubernetes/ssl/kube-ca.pem --tls-private-key-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224-key.pem --event-qps=0 --address=0.0.0.0 --cgroups-per-qos=True --pod-infra-container-image=rancher/mirrored-pause:3.7 --root-dir=/var/lib/kubelet --container-runtime=remote --make-iptables-util-chains=true --authorization-mode=Webhook --resolv-conf=/etc/resolv.conf --cloud-provider= --container-runtime-endpoint=unix:///var/run/cri-dockerd.sock --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 --authentication-token-webhook=true --anonymous-auth=false --read-only-port=0 --volume-plugin-dir=/var/lib/kubelet/volumeplugins --protect-kernel-defaults=true --feature-gates=RotateKubeletServerCertificate=true --cluster-dns=10.43.0.10 --fail-swap-on=false --hostname-override=ip-172-31-4-224 --kubeconfig=/etc/kubernetes/ssl/kubecfg-kube-node.yaml --cluster-domain=cluster.local --tls-cert-file=/etc/kubernetes/ssl/kube-kubelet-172-31-4-224.pem --streaming-connection-idle-timeout=30m --cgroup-driver=cgroupfs --resolv-conf=/run/systemd/resolve/resolv.conf +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[]?.kind=="ServiceAccount" and .subjects[]?.name=="default") or (.subjects[]?.kind=="Group" and .subjects[]?.name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[]? != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +### 5.1.9 Minimize access to create persistent volumes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to PersistentVolume objects in the cluster. + +### 5.1.10 Minimize access to the proxy sub-resource of nodes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the proxy sub-resource of node objects. + +### 5.1.11 Minimize access to the approval sub-resource of certificatesigningrequests objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the approval sub-resource of certificatesigningrequest objects. + +### 5.1.12 Minimize access to webhook configuration objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the validatingwebhookconfigurations or mutatingwebhookconfigurations objects + +### 5.1.13 Minimize access to the service account token creation (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the token sub-resource of serviceaccount objects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +### 5.2.7 Minimize the admission of root containers (Manual) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md new file mode 100644 index 00000000000..431d87bb1ed --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-hardening-guide.md @@ -0,0 +1,273 @@ +--- +title: RKE2 加固指南 +--- + + + + + +本文档提供了针对生产环境的 RKE2 集群进行加固的具体指导，以便在使用 Rancher 部署之前进行配置。它概述了满足信息安全中心（Center for Information Security, CIS）Kubernetes benchmark controls 制所需的配置和控制。 + +:::note +这份加固指南描述了如何确保你集群中的节点安全。我们建议你在安装 Kubernetes 之前遵循本指南。 +::: + +此加固指南适用于 RKE2 集群，并与以下版本的 CIS Kubernetes Benchmark、Kubernetes 和 Rancher 相关联： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|------------------------------| +| Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25 至 v1.26 | + +:::note +- 在 Benchmark v1.24 及更高版本中，由于新的文件权限要求（600 而不是 644），一些检查 ID 可能会失败。受影响的检查 ID 包括：`1.1.1`, `1.1.3`, `1.1.5`, `1.1.7`, `1.1.13`, `1.1.15`, `1.1.17`, `4.1.3`, `4.1.5` 和 `4.1.9`。 +- 在 Benchmark v1.7 中，不再需要 `--protect-kernel-defaults` (4.2.6) 参数，并已被 CIS 删除。 +::: + +有关如何评估加固的 RKE2 集群与官方 CIS benchmark 的更多细节，请参考特定 Kubernetes 和 CIS benchmark 版本的 RKE2 自我评估指南。 + +RKE2 在不需要修改的情况下通过了许多 Kubernetes CIS controls，因为它默认应用了几个安全缓解措施。然而，有一些值得注意的例外情况，需要手动干预才能完全符合 CIS Benchmark 要求： + +1. RKE2 不会修改主机操作系统。因此，作为运维人员，你必须进行一些主机级别的修改。 +2. 某些 CIS controls 对于网络策略和 Pod 安全标准（或 RKE2 v1.25 之前的 Pod 安全策略 (PSP)）将限制集群的功能。你必须选择让 RKE2 为你配置这些功能。为了确保满足这些要求，可以启动 RKE2 并设置 profile 标志为 `cis-1.23`（适用于 v1.25 及更新版本）或 `cis-1.6`（适用于 v1.24 及更早版本）。 + +## 主机级别要求 + +主机级要求有两个方面：内核参数和 etcd 进程/目录配置。这些在本节中进行了概述。 + +### 确保 `protect-kernel-defaults` 已经设置 + + + + +自 CIS benchmark 1.7 开始，不再需要`protect-kernel-defaults`。 + + + + +这是一个 kubelet 标志，如果所需的内核参数未设置或设置为与 kubelet 的默认值不同的值，将导致 kubelet 退出。 + +可以在 Rancher 的集群配置中设置 `protect-kernel-defaults` 标志。 + +```yaml +spec: + rkeConfig: + machineSelectorConfig: + - config: + protect-kernel-defaults: true +``` + + + + +### 设置内核参数 + +建议为集群中的所有节点类型设置以下 `sysctl` 配置。在 `/etc/sysctl.d/90-kubelet.conf` 中设置以下参数： + +```ini +vm.panic_on_oom=0 +vm.overcommit_memory=1 +kernel.panic=10 +kernel.panic_on_oops=1 +``` + +运行 `sudo sysctl -p /etc/sysctl.d/90-kubelet.conf` 以启用设置。 + +### 确保 etcd 配置正确 + +CIS Benchmark 要求 etcd 数据目录由 `etcd` 用户和组拥有。这意味着 etcd 进程必须以主机级别的 `etcd` 用户身份运行。为了实现这一点，在使用有效的 `cis-1.xx` 配置文件启动 RKE2 时，RKE2 会采取以下几个步骤： + +1. 检查主机上是否存在 `etcd` 用户和组。如果不存在，则显示错误并退出。 +2. 使用 `etcd` 作为用户和组所有者创建 etcd 的数据目录。 +3. 通过适当设置 etcd 静态 Pod 的 `SecurityContext`，确保 etcd 进程以 `etcd` 用户和组的身份运行。 + +为满足上述要求，你必须执行以下操作： + +#### 创建 etcd 用户 + +在某些 Linux 发行版中，`useradd` 命令不会创建组。下面包含了 `-U` 标志来解决这个问题。这个标志告诉 `useradd` 创建一个与用户同名的组。 + +```bash +sudo useradd -r -c "etcd user" -s /sbin/nologin -M etcd -U +``` + +## Kubernetes 运行时要求 + +通过 CIS Benchmark 测试的运行时要求主要集中在 Pod 安全、网络策略和内核参数上。当使用有效的 `cis-1.xx` 配置文件时，大部分都会被 RKE2 自动处理，但需要一些额外的运维人员干预是。本节概述了这些内容。 + +### Pod 安全 + +RKE2 始终以一定程度的 Pod 安全性运行。 + + + + +在 v1.25 及更高版本中，[Pod 安全准入(PSAs)](https://kubernetes.io/docs/concepts/security/pod-security-admission/)用于保证 pod 安全。 + +以下是确保加固 RKE2 通过 Rancher 中提供的 CIS v1.23 加固配置文件 `rke2-cis-1.7-hardened` 所需的最低配置。 + +```yaml +spec: + defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted + rkeConfig: + machineSelectorConfig: + - config: + profile: cis-1.23 +``` + +当同时设置了 `defaultPodSecurityAdmissionConfigurationTemplateName` 和 `profile` 标志时，Rancher 和 RKE2 会执行以下操作： + +1. 检查是否已满足主机级要求。如果未满足，RKE2 将以致命错误退出，并描述未满足的要求。 +2. 应用允许群集传递关联 controls 的网络策略。 +3. 配置 Pod 安全准入控制器（PSA）使用 PSA 配置模板 `rancher-restricted`，以在所有命名空间中强制执行受限模式，除了模板豁免列表中的命名空间。 + 这些命名空间被豁免，以允许系统 Pod 在没有限制的情况下运行，这是集群正常运行所必需的。 + +:::note +如果你打算将一个 RKE 集群导入到 Rancher 中，请参考[文档](../../../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/psa-config-templates.md)了解如何配置 PSA 以豁免 Rancher system 命名空间。 +::: + + + + + +在 Kubernetes v1.24 及更早版本中，`PodSecurityPolicy` 准入控制器始终是启用的。 + +以下是确保 RKE2 加固以通过 Rancher 中提供的 CIS v1.23 加固配置文件 `rke2-cis-1.23-hardened` 所需的最低配置。 + +:::note +在下面的示例中，配置文件设置为`cis-1.6`，这是在上游 RKE2 中定义的值，但集群实际上配置为传递 CIS v1.23 加固配置文件 +::: + +```yaml +spec: + defaultPodSecurityPolicyTemplateName: restricted-noroot + rkeConfig: + machineSelectorConfig: + - config: + profile: cis-1.6 +``` + + +当同时设置了 `defaultPodSecurityPolicyTemplateName` 和 `profile` 标志时，Rancher 和 RKE2 会执行以下操作： + +1. 检查是否已满足主机级要求。如果未满足，RKE2 将以致命错误退出，并描述未满足的要求。 +2. 应用网络策略，以确保集群通过相关的 controls 要求。 +3. 配置运行时 Pod 安全策略，以确保集群通过相关的 controls 要求。 + + + + +:::note +Kubernetes control plane 组件以及关键的附加组件，如 CNI、DNS 和 Ingress，都作为 `kube-system` 命名空间中的 Pod 运行。因此，这个命名空间的限制政策较少，从而使这些组件能够正常运行。 +::: + +### 网络策略 + +当使用有效的 `cis-1.xx` 配置文件运行时，RKE2 将设置适当的 `NetworkPolicies`，以满足 Kubernetes 内置命名空间的 CIS Benchmark 要求。这些命名空间包括：`kube-system`、`kube-public`、`kube-node-lease` 和 `default`。 + +所使用的 `NetworkPolicy` 仅允许同一命名空间内的 Pod 相互通信。值得注意的例外是允许 DNS 请求进行解析。 + +:::note +运维人员必须像管理其他命名空间一样管理额外创建的命名空间的网络策略。 +::: + +### 配置 `default` service account + +**将 `default` service accountsSet 的 `automountServiceAccountToken` 设置为 `false`** + +Kubernetes 提供了一个 `default` service account，用于集群工作负载，在 pod 没有分配特定 service account 时使用。如果需要从 pod 访问 Kubernetes API，则应为该 pod 创建一个特定的 service account，并授予该 service account 权限。`default` service account 应配置为不提供 service account 令牌，并且不具有任何明确的权限分配。 + +对于标准的 RKE2 安装中的每个命名空间，包括 `default` 和 `kube-system`，`default` service account 必须包含此值： + +```yaml +automountServiceAccountToken: false +``` + +对于由集群操作员创建的命名空间，可以使用以下脚本和配置文件来配置 `default` service account。 + +以下配置必须保存到一个名为 `account_update.yaml` 的文件中。 + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: default +automountServiceAccountToken: false +``` + +创建一个名为 `account_update.sh` 的 bash 脚本文件。确保运行 `sudo chmod +x account_update.sh` 命令，以便脚本具有执行权限。 + +```bash +#!/bin/bash -e + +for namespace in $(kubectl get namespaces -A -o=jsonpath="{.items[*]['metadata.name']}"); do + echo -n "Patching namespace $namespace - " + kubectl patch serviceaccount default -n ${namespace} -p "$(cat account_update.yaml)" +done +``` + +执行此脚本以将 `account_update.yaml` 配置应用到所有命名空间中的 `default` service account。 + +### API server 审计配置 + +CIS 要求 1.2.19 至 1.2.22 与为 API server 配置审计日志有关。当 RKE2 在设置配置文件标志的情况下启动时，它将自动在 API server 中配置加固的 `--audit-log-` 参数以通过这些 CIS 检查。 + +RKE2 的默认审计策略被配置为不记录 API server 中的请求。这样做是为了允许集群操作员灵活地定制符合其审计要求和需求的审计策略，因为这些策略是针对每个用户的环境和政策而特定的。 + +当使用 `profile` 标志启动 RKE2 时，RKE2 会创建一个默认的审计策略。该策略定义在 `/etc/rancher/rke2/audit-policy.yaml` 中。 + +```yaml +apiVersion: audit.k8s.io/v1 +kind: Policy +metadata: + creationTimestamp: null +rules: +- level: None +``` + +## 加固的 RKE2 模板配置参考 + +参考模板配置用于在 Rancher 中创建加固的 RKE2 自定义集群。该参考不包括其他必需的**集群配置**指令，这些指令会根据你的环境而有所不同。 + + + + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: # 定义集群名称 +spec: + defaultPodSecurityAdmissionConfigurationTemplateName: rancher-restricted + kubernetesVersion: # 定义 RKE2 版本 + rkeConfig: + machineSelectorConfig: + - config: + profile: cis-1.23 +``` + + + + +```yaml +apiVersion: provisioning.cattle.io/v1 +kind: Cluster +metadata: + name: # 定义集群名称 +spec: + defaultPodSecurityPolicyTemplateName: restricted-noroot + kubernetesVersion: # 定义 RKE2 版本 + rkeConfig: + machineSelectorConfig: + - config: + profile: cis-1.6 + protect-kernel-defaults: true +``` + + + + +## 结论 + +如果你按照本指南操作，由 Rancher 提供的 RKE2 自定义集群将配置为通过 CIS Kubernetes Benchmark 测试。你可以查看我们的 RKE2 自我评估指南，了解我们是如何验证每个 benchmarks 的，并且你可以在你的集群上执行相同的操作。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md index 83cac7d6e59..05ec2b4aef5 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.23-k8s-v1.23.md @@ -1,30 +1,34 @@ --- -title: RKE2 Self-Assessment Guide - CIS Benchmark v1.23 - K8s v1.23 +title: RKE2 自我评估指南 - CIS Benchmark v1.23 - K8s v1.23 --- -This document is a companion to the [RKE2 Hardening Guide](../../../../pages-for-subheaders/rke2-hardening-guide.md), which provides prescriptive guidance on how to harden RKE2 clusters that are running in production and managed by Rancher. This benchmark guide helps you evaluate the security of a hardened cluster against each control in the CIS Kubernetes Benchmark. + + + -This guide corresponds to the following versions of Rancher, CIS Benchmarks, and Kubernetes: +本文档是 [RKE2 加固指南](rke2-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE2 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 -| Rancher Version | CIS Benchmark Version | Kubernetes Version | +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | |-----------------|-----------------------|--------------------| | Rancher v2.7 | Benchmark v1.23 | Kubernetes v1.23 | -This guide walks through the various controls and provide updated example commands to audit compliance in Rancher created clusters. Because Rancher and RKE2 install Kubernetes services as Docker containers, many of the control verification checks in the CIS Kubernetes Benchmark don't apply. These checks will return a result of `Not Applicable`. +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE2 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 -This document is for Rancher operators, security teams, auditors and decision makers. +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 -For more information about each control, including detailed descriptions and remediations for failing tests, refer to the corresponding section of the CIS Kubernetes Benchmark v1.23. You can download the benchmark, after creating a free account, at [Center for Internet Security (CIS)](https://www.cisecurity.org/benchmark/kubernetes/). +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.23 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 -## Testing Methodology +## 测试方法 -RKE2 launches control plane components as static pods, managed by the kubelet, and uses containerd as the container runtime. Configuration is defined by arguments passed to the container at the time of initialization or via configuration file. +RKE2 将 control plane 组件作为静态 Pod 启动，由 kubelet 管理，并使用 containerd 作为容器运行时。配置是由初始化时或通过配置文件传递给容器的参数定义的。 -Where control audits differ from the original CIS benchmark, the audit commands specific to Rancher are provided for testing. When performing the tests, you will need access to the command line on the hosts of all RKE2 nodes. The commands also make use of the [kubectl](https://kubernetes.io/docs/tasks/tools/) (with a valid configuration file) and [jq](https://stedolan.github.io/jq/) tools, which are required in the testing and evaluation of test results. +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE2 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 :::note -This guide only covers `automated` (previously called `scored`) tests. +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 ::: diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md new file mode 100644 index 00000000000..f14207e929d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.24-k8s-v1.24.md @@ -0,0 +1,3202 @@ +--- +title: RKE2 自我评估指南 - CIS Benchmark v1.24 - K8s v1.24 +--- + + + + + +本文档是 [RKE2 加固指南](rke2-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE2 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.24 | Kubernetes v1.24 | + +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE2 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.24 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +RKE2 将 control plane 组件作为静态 Pod 启动，由 kubelet 管理，并使用 containerd 作为容器运行时。配置是由初始化时或通过配置文件传递给容器的参数定义的。 + +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE2 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 644 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %a /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' +``` + +**Expected Result**: + +```console +'600' is present +``` + +**Returned Value**: + +```console +644 +``` + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' +``` + +**Expected Result**: + +```console +'permissions' is equal to '600' +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 644 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 644 /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' +``` + +**Expected Result**: + +```console +'644' is equal to '644' +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=644 +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root root:root +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/db/etcd +``` + +**Expected Result**: + +```console +permissions has permissions 700, expected 700 or more restrictive +``` + +**Returned Value**: + +```console +permissions=700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/db/etcd +``` + +**Expected Result**: + +```console +'etcd:etcd' is present +``` + +**Returned Value**: + +```console +etcd:etcd +``` + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/admin.conf + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/cred/admin.kubeconfig +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/admin.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 scheduler + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root scheduler + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 controllermanager + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/cred/controller.kubeconfig +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/server/cred/controller.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/controller.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /var/lib/rancher/rke2/server/tls/ + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/tls +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Automated) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 644 /var/lib/rancher/rke2/server/tls/*.crt + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.crt +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /var/lib/rancher/rke2/server/tls/*.key + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.key +``` + +**Expected Result**: + +```console +'permissions' is equal to '600' +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.4 Ensure that the --kubelet-https argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the --kubelet-https parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-https' is present OR '--kubelet-https' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.5 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.6 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.7 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.9 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.10 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.12 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.13 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.14 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.15 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.16 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.17 Ensure that the --secure-port argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.18 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.19 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.20 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.21 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.22 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxsize' is greater or equal to 100 +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.24 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.25 Ensure that the --request-timeout argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true root 1017917 2489 83 16:16 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/readyz root 1017945 2489 75 16:16 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/livez +``` + +### 1.2.26 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.27 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.28 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.29 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.30 Ensure that the --encryption-provider-config argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 2548 2489 10 Sep11 ? 02:10:01 kube-apiserver --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction,PodSecurityPolicy --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.32 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 + +### 1.2.33 Ensure that encryption providers are appropriately configured (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit:** + +```bash +/bin/sh -c 'if grep aescbc /var/lib/rancher/rke2/server/cred/encryption-config.json; then echo 0; fi' +``` + +**Expected Result**: + +```console +'0' is present +``` + +**Returned Value**: + +```console +{"kind":"EncryptionConfiguration","apiVersion":"apiserver.config.k8s.io/v1","resources":[{"resources":["secrets"],"providers":[{"aescbc":{"keys":[{"name":"aescbckey","secret":"akmMjUAq94YvsUftJpiA3b+9SClu0ESPBeckAI7KZBY="}]}},{"identity":{}}]}]} 0 +``` + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 2743 2649 2 Sep11 ? 00:28:36 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2707 2593 0 Sep11 ? 00:06:20 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 2707 2593 0 Sep11 ? 00:06:20 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-10-113 ETCD_UNSUPPORTED_ARCH= NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=560b915d9afb672c20c2c1e8664bdf8f FILE_HASH=166aebdce42ff62fcdd2cefc9e7ed8f7c5b562d219ca6afec8f73adc654f65e7 HOME=/ +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-10-113 ETCD_UNSUPPORTED_ARCH= NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=560b915d9afb672c20c2c1e8664bdf8f FILE_HASH=166aebdce42ff62fcdd2cefc9e7ed8f7c5b562d219ca6afec8f73adc654f65e7 HOME=/ +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Audit Config:** + +```bash +cat /var/lib/rancher/rke2/server/db/etcd/config +``` + +**Expected Result**: + +```console +'ETCD_TRUSTED_CA_FILE' is present OR '{.peer-transport-security.trusted-ca-file}' is equal to '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-10-113 ETCD_UNSUPPORTED_ARCH= NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=560b915d9afb672c20c2c1e8664bdf8f FILE_HASH=166aebdce42ff62fcdd2cefc9e7ed8f7c5b562d219ca6afec8f73adc654f65e7 HOME=/ +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep | grep -o audit-policy-file +``` + +**Expected Result**: + +```console +'audit-policy-file' is equal to 'audit-policy-file' +``` + +**Returned Value**: + +```console +audit-policy-file +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'600' is present +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit Script:** `check_cafile_permissions.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +CAFILE=/node$CAFILE +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_permissions.sh +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit Script:** `check_cafile_ownership.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +CAFILE=/node$CAFILE +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_ownership.sh +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.6 Ensure that the --protect-kernel-defaults argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `protectKernelDefaults` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--protect-kernel-defaults=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--protect-kernel-defaults' is equal to 'true' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.7 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.makeIPTablesUtilChains}' is present OR '{.makeIPTablesUtilChains}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.8 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +### 4.2.9 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Automated) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.eventRecordQPS}' is present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.10 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2291 2246 4 Sep11 ? 00:50:01 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-10-113 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=6685035f-32be-4d1b-a06d-7ea5f42467f5 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.11 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.12 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** pass + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.featureGates.RotateKubeletServerCertificate}' is present OR '{.featureGates.RotateKubeletServerCertificate}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.13 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{range .tlsCipherSuites[:]}{}{','}{end}' is present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Automated) + + +**Result:** pass + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +**Audit Script:** `check_for_default_sa.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +count_sa=$(kubectl get serviceaccounts --all-namespaces -o json | jq -r '.items[] | select(.metadata.name=="default") | select((.automountServiceAccountToken == null) or (.automountServiceAccountToken == true))' | jq .metadata.namespace | wc -l) +if [[ ${count_sa} -gt 0 ]]; then + echo "false" + exit +fi + +for ns in $(kubectl get ns --no-headers -o custom-columns=":metadata.name") +do + for result in $(kubectl get clusterrolebinding,rolebinding -n $ns -o json | jq -r '.items[] | select((.subjects[]?.kind=="ServiceAccount" and .subjects[]?.name=="default") or (.subjects[]?.kind=="Group" and .subjects[]?.name=="system:serviceaccounts"))' | jq -r '"\(.roleRef.kind),\(.roleRef.name)"') + do + read kind name <<<$(IFS=","; echo $result) + resource_count=$(kubectl get $kind $name -n $ns -o json | jq -r '.rules[] | select(.resources[]? != "podsecuritypolicies")' | wc -l) + if [[ ${resource_count} -gt 0 ]]; then + echo "false" + exit + fi + done +done + + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_default_sa.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +**Audit:** + +```bash +kubectl get psp global-restricted-psp && kubectl get psp global-restricted-psp -o json | jq -r ".spec.runAsUser.rule" || kubectl get psp restricted-noroot-psp && kubectl get psp restricted-noroot-psp -o json | jq -r ".spec.runAsUser.rule" +``` + +**Expected Result**: + +```console +'MustRunAsNonRoot' is equal to 'MustRunAsNonRoot' +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ Error from server (NotFound): podsecuritypolicies.policy "global-restricted-psp" not found Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES restricted-noroot-psp false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,csi,persistentVolumeClaim Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ MustRunAsNonRoot +``` + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostPID == null) or (.spec.hostPID == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=5 +``` + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostIPC == null) or (.spec.hostIPC == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=5 +``` + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.hostNetwork == null) or (.spec.hostNetwork == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=2 +``` + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=4 +``` + +### 5.2.7 Minimize the admission of root containers (Automated) + + +**Result:** pass + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +**Audit:** + +```bash +kubectl get psp -o json | jq .items[] | jq -r 'select((.spec.allowPrivilegeEscalation == null) or (.spec.allowPrivilegeEscalation == false))' | jq .metadata.name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ --count=4 +``` + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Automated) + + +**Result:** pass + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +**Audit:** + +```bash +kubectl get psp global-restricted-psp && kubectl get psp global-restricted-psp -o json | jq -r .spec.requiredDropCapabilities[] || kubectl get psp restricted-noroot-psp && kubectl get psp restricted-noroot-psp -o json | jq -r .spec.requiredDropCapabilities[] +``` + +**Expected Result**: + +```console +'ALL' is equal to 'ALL' +``` + +**Returned Value**: + +```console +Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ Error from server (NotFound): podsecuritypolicies.policy "global-restricted-psp" not found Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ NAME PRIV CAPS SELINUX RUNASUSER FSGROUP SUPGROUP READONLYROOTFS VOLUMES restricted-noroot-psp false RunAsAny MustRunAsNonRoot MustRunAs MustRunAs false configMap,emptyDir,projected,secret,downwardAPI,csi,persistentVolumeClaim Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+ ALL +``` + +### 5.2.9 Minimize the admission of containers with added capabilities (Automated) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports Network Policies (Automated) + + +**Result:** pass + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +**Audit:** + +```bash +kubectl get pods --all-namespaces --selector='k8s-app in (calico-node, canal, cilium)' -o name | wc -l | xargs -I {} echo '--count={}' +``` + +**Expected Result**: + +```console +'count' is greater than 0 +``` + +**Returned Value**: + +```console +--count=1 +``` + +### 5.3.2 Ensure that all Namespaces have Network Policies defined (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +**Audit Script:** `check_for_rke2_network_policies.sh` + +```bash +#!/bin/bash + +set -eE + +handle_error() { + echo "false" +} + +trap 'handle_error' ERR + +for namespace in kube-system kube-public default; do + policy_count=$(/var/lib/rancher/rke2/bin/kubectl get networkpolicy -n ${namespace} -o json | jq -r '.items | length') + if [ ${policy_count} -eq 0 ]; then + echo "false" + exit + fi +done + +echo "true" + +``` + +**Audit Execution:** + +```bash +./check_for_rke2_network_policies.sh +``` + +**Expected Result**: + +```console +'true' is equal to 'true' +``` + +**Returned Value**: + +```console +true +``` + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md new file mode 100644 index 00000000000..a51c617eea8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/hardening-guides/rke2-hardening-guide/rke2-self-assessment-guide-with-cis-v1.7-k8s-v1.25-v1.26-v1.27.md @@ -0,0 +1,2967 @@ +--- +title: RKE2 自我评估指南 - CIS Benchmark v1.7 - K8s v1.25/v1.26/v1.27 +--- + + + + + +本文档是 [RKE2 加固指南](rke2-hardening-guide.md)的配套文档，该指南提供了关于如何加固正在生产环境中运行并由 Rancher 管理的 RKE2 集群的指导方针。本 benchmark 指南可帮助你根据 CIS Kubernetes Benchmark 中的每个 control 来评估加固集群的安全性。 + +本指南对应以下版本的 Rancher、CIS Benchmarks 和 Kubernetes： + +| Rancher 版本 | CIS Benchmark 版本 | Kubernetes 版本 | +|-----------------|-----------------------|--------------------| +| Rancher v2.7 | Benchmark v1.7 | Kubernetes v1.25/v1.26/v1.27 | + +本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群中的合规性。由于 Rancher 和 RKE2 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多 control 验证检查不适用。这些检查将返回 `Not Applicable` 的结果。 + +本文档适用于 Rancher 运维人员、安全团队、审计员和决策者。 + +有关每个 control 的更多信息，包括详细描述和未通过测试的补救措施，请参考 CIS Kubernetes Benchmark v1.7 的相应部分。你可以在[互联网安全中心 (CIS)](https://www.cisecurity.org/benchmark/kubernetes/)创建免费账户后下载 benchmark。 + +## 测试方法 + +RKE2 将 control plane 组件作为静态 Pod 启动，由 kubelet 管理，并使用 containerd 作为容器运行时。配置是由初始化时或通过配置文件传递给容器的参数定义的。 + +在 control 审计与原始 CIS benchmark 不同时，提供了针对 Rancher 的特定审计命令以进行测试。在执行测试时，你将需要访问所有 RKE2 节点主机上的命令行。这些命令还使用了 [kubectl](https://kubernetes.io/docs/tasks/tools/)（带有有效的配置文件）和 [jq](https://stedolan.github.io/jq/) 工具，在测试和评估测试结果时这些工具是必需的。 + +:::note + +本指南仅涵盖 `automated`（之前称为 `scored`）测试。 + +::: + +### Controls + +## 1.1 Control Plane Node Configuration Files +### 1.1.1 Ensure that the API server pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the +control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.2 Ensure that the API server pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.3 Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.4 Ensure that the controller manager pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.5 Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c permissions=%a /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.6 Ensure that the scheduler pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.7 Ensure that the etcd pod specification file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then find /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml -name '*etcd*' | xargs stat -c permissions=%a; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.8 Ensure that the etcd pod specification file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; then stat -c %U:%G /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.9 Ensure that the Container Network Interface file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c permissions=%a find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c permissions=%a +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=644 +``` + +### 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root + +**Audit:** + +```bash +ps -fC ${kubeletbin:-kubelet} | grep -- --cni-conf-dir || echo "/etc/cni/net.d" | sed 's%.*cni-conf-dir[= ]\([^ ]*\).*%\1%' | xargs -I{} find {} -mindepth 1 | xargs --no-run-if-empty stat -c %U:%G find /var/lib/cni/networks -type f 2> /dev/null | xargs --no-run-if-empty stat -c %U:%G +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root root:root +``` + +### 1.1.11 Ensure that the etcd data directory permissions are set to 700 or more restrictive (Automated) + + +**Result:** pass + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). For example, +chmod 700 /var/lib/etcd + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/db/etcd +``` + +**Expected Result**: + +```console +permissions has permissions 700, expected 700 or more restrictive +``` + +**Returned Value**: + +```console +permissions=700 +``` + +### 1.1.12 Ensure that the etcd data directory ownership is set to etcd:etcd (Automated) + + +**Result:** Not Applicable + +**Remediation:** +On the etcd server node, get the etcd data directory, passed as an argument --data-dir, +from the command 'ps -ef | grep etcd'. +Run the below command (based on the etcd data directory found above). +For example, chown etcd:etcd /var/lib/etcd + +### 1.1.13 Ensure that the admin.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chmod 600 /etc/kubernetes/admin.conf + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/cred/admin.kubeconfig +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.14 Ensure that the admin.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, chown root:root /etc/kubernetes/admin.conf + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/admin.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.15 Ensure that the scheduler.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'600' is present +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.16 Ensure that the scheduler.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/scheduler.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.17 Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod 600 /var/lib/rancher/rke2/server/cred/controller.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/server/cred/controller.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/server/cred/controller.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'600' is present +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 1.1.18 Ensure that the controller-manager.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown root:root /var/lib/rancher/rke2/server/cred/controller.kubeconfig + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/cred/controller.kubeconfig +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.19 Ensure that the Kubernetes PKI directory and file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chown -R root:root /etc/kubernetes/pki/ + +**Audit:** + +```bash +stat -c %U:%G /var/lib/rancher/rke2/server/tls +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 1.1.20 Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /var/lib/rancher/rke2/server/tls/*.crt + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.crt +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 permissions=644 +``` + +### 1.1.21 Ensure that the Kubernetes PKI key file permissions are set to 600 (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the control plane node. +For example, +chmod -R 600 /var/lib/rancher/rke2/server/tls/*.key + +**Audit:** + +```bash +stat -c permissions=%a /var/lib/rancher/rke2/server/tls/*.key +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 permissions=600 +``` + +## 1.2 API Server +### 1.2.1 Ensure that the --anonymous-auth argument is set to false (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--anonymous-auth=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +### 1.2.2 Ensure that the --token-auth-file parameter is not set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and configure alternate mechanisms for authentication. Then, +edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the --token-auth-file= parameter. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--token-auth-file' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.3 Ensure that the --DenyServiceExternalIPs is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and remove the `DenyServiceExternalIPs` +from enabled admission plugins. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'DenyServiceExternalIPs' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.4 Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the +apiserver and kubelets. Then, edit API server pod specification file +/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the +kubelet client certificate and key parameters as below. +--kubelet-client-certificate= +--kubelet-client-key= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-client-certificate' is present AND '--kubelet-client-key' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.5 Ensure that the --kubelet-certificate-authority argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and setup the TLS connection between +the apiserver and kubelets. Then, edit the API server pod specification file +/var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml on the control plane node and set the +--kubelet-certificate-authority parameter to the path to the cert file for the certificate authority. +--kubelet-certificate-authority= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--kubelet-certificate-authority' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.6 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to values other than AlwaysAllow. +One such example could be as below. +--authorization-mode=RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.7 Ensure that the --authorization-mode argument includes Node (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes Node. +--authorization-mode=Node,RBAC + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'Node' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.8 Ensure that the --authorization-mode argument includes RBAC (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --authorization-mode parameter to a value that includes RBAC, +for example `--authorization-mode=Node,RBAC`. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--authorization-mode' has 'RBAC' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.9 Ensure that the admission control plugin EventRateLimit is set (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and set the desired limits in a configuration file. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +and set the below parameters. +--enable-admission-plugins=...,EventRateLimit,... +--admission-control-config-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'EventRateLimit' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.10 Ensure that the admission control plugin AlwaysAdmit is not set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and either remove the --enable-admission-plugins parameter, or set it to a +value that does not include AlwaysAdmit. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' does not have 'AlwaysAdmit' OR '--enable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.11 Ensure that the admission control plugin AlwaysPullImages is set (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +AlwaysPullImages. +--enable-admission-plugins=...,AlwaysPullImages,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'AlwaysPullImages' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.12 Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used (Manual) + + +**Result:** warn + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to include +SecurityContextDeny, unless PodSecurityPolicy is already in place. +--enable-admission-plugins=...,SecurityContextDeny,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'SecurityContextDeny' OR '--enable-admission-plugins' has 'PodSecurityPolicy' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.13 Ensure that the admission control plugin ServiceAccount is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the documentation and create ServiceAccount objects as per your environment. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and ensure that the --disable-admission-plugins parameter is set to a +value that does not include ServiceAccount. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true root 1018768 2419 99 16:17 ? 00:00:00 kubectl get --server=https://localhost:6443/ --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/readyz +``` + +### 1.2.14 Ensure that the admission control plugin NamespaceLifecycle is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --disable-admission-plugins parameter to +ensure it does not include NamespaceLifecycle. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--disable-admission-plugins' is present OR '--disable-admission-plugins' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.15 Ensure that the admission control plugin NodeRestriction is set (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure NodeRestriction plug-in on kubelets. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --enable-admission-plugins parameter to a +value that includes NodeRestriction. +--enable-admission-plugins=...,NodeRestriction,... + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--enable-admission-plugins' has 'NodeRestriction' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.16 Ensure that the --secure-port argument is not set to 0 - NoteThis recommendation is obsolete and will be deleted per the consensus process (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and either remove the --secure-port parameter or +set it to a different (non-zero) desired port. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--secure-port' is greater than 0 OR '--secure-port' is not present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.17 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.18 Ensure that the --audit-log-path argument is set (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-path parameter to a suitable path and +file where you would like audit logs to be written, for example, +--audit-log-path=/var/log/apiserver/audit.log + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-path' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.19 Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxage parameter to 30 +or as an appropriate number of days, for example, +--audit-log-maxage=30 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxage' is greater or equal to 30 +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.20 Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxbackup parameter to 10 or to an appropriate +value. For example, +--audit-log-maxbackup=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-log-maxbackup' is greater or equal to 10 +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.21 Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --audit-log-maxsize parameter to an appropriate size in MB. +For example, to set it as 100 MB, --audit-log-maxsize=100 + +### 1.2.22 Ensure that the --request-timeout argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +and set the below parameter as appropriate and if needed. +For example, --request-timeout=300s + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--request-timeout' is not present OR '--request-timeout' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.23 Ensure that the --service-account-lookup argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--service-account-lookup=true +Alternatively, you can delete the --service-account-lookup parameter from this file so +that the default takes effect. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-lookup' is not present OR '--service-account-lookup' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.24 Ensure that the --service-account-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --service-account-key-file parameter +to the public key file for service accounts. For example, +--service-account-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-key-file' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.25 Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate and key file parameters. +--etcd-certfile= +--etcd-keyfile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-certfile' is present AND '--etcd-keyfile' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.26 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the TLS certificate and private key file parameters. +--tls-cert-file= +--tls-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.27 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection on the apiserver. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the client certificate authority file. +--client-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.28 Ensure that the --etcd-cafile argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and set up the TLS connection between the apiserver and etcd. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the etcd certificate authority file parameter. +--etcd-cafile= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--etcd-cafile' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.29 Ensure that the --encryption-provider-config argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +Then, edit the API server pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-apiserver.yaml +on the control plane node and set the --encryption-provider-config parameter to the path of that file. +For example, --encryption-provider-config= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--encryption-provider-config' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.2.30 Ensure that encryption providers are appropriately configured (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and configure a EncryptionConfig file. +In this file, choose aescbc, kms or secretbox as the encryption provider. + +**Audit:** + +```bash +ENCRYPTION_PROVIDER_CONFIG=$(ps -ef | grep kube-apiserver | grep -- --encryption-provider-config | sed 's%.*encryption-provider-config[= ]\([^ ]*\).*%\1%') if test -e $ENCRYPTION_PROVIDER_CONFIG; then grep -A1 'providers:' $ENCRYPTION_PROVIDER_CONFIG | tail -n1 | grep -o "[A-Za-z]*" | sed 's/^/provider=/'; fi +``` + +**Expected Result**: + +```console +'provider' is present +``` + +### 1.2.31 Ensure that the API Server only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml +on the control plane node and set the below parameter. +--tls-cipher-suites=TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, +TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, +TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, +TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, +TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA, +TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384 +Not Applicable. + +## 1.3 Controller Manager +### 1.3.1 Ensure that the --terminated-pod-gc-threshold argument is set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --terminated-pod-gc-threshold to an appropriate threshold, +for example, --terminated-pod-gc-threshold=10 + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--terminated-pod-gc-threshold' is present +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.2 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.3 Ensure that the --use-service-account-credentials argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node to set the below parameter. +--use-service-account-credentials=true + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--use-service-account-credentials' is not equal to 'false' +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.4 Ensure that the --service-account-private-key-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --service-account-private-key-file parameter +to the private key file for service accounts. +--service-account-private-key-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--service-account-private-key-file' is present +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.5 Ensure that the --root-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --root-ca-file parameter to the certificate bundle file`. +--root-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--root-ca-file' is present +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 1.3.6 Ensure that the RotateKubeletServerCertificate argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and set the --feature-gates parameter to include RotateKubeletServerCertificate=true. +--feature-gates=RotateKubeletServerCertificate=true + +### 1.3.7 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Controller Manager pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-controller-manager.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-controller-manager | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +## 1.4 Scheduler +### 1.4.1 Ensure that the --profiling argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml file +on the control plane node and set the below parameter. +--profiling=false + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--profiling' is equal to 'false' +``` + +**Returned Value**: + +```console +root 2645 2538 0 Sep11 ? 00:05:26 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 +``` + +### 1.4.2 Ensure that the --bind-address argument is set to 127.0.0.1 (Automated) + + +**Result:** pass + +**Remediation:** +Edit the Scheduler pod specification file /var/lib/rancher/rke2/agent/pod-manifests/kube-scheduler.yaml +on the control plane node and ensure the correct value for the --bind-address parameter + +**Audit:** + +```bash +/bin/ps -ef | grep kube-scheduler | grep -v grep +``` + +**Expected Result**: + +```console +'--bind-address' is equal to '127.0.0.1' OR '--bind-address' is not present +``` + +**Returned Value**: + +```console +root 2645 2538 0 Sep11 ? 00:05:26 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-scheduler --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 +``` + +## 2 Etcd Node Configuration +### 2.1 Ensure that the --cert-file and --key-file arguments are set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the etcd service documentation and configure TLS encryption. +Then, edit the etcd pod specification file /etc/kubernetes/manifests/etcd.yaml +on the master node and set the below parameters. +--cert-file= +--key-file= +Not Applicable. + +### 2.2 Ensure that the --client-cert-auth argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and set the below parameter. +--client-cert-auth="true" +Not Applicable. + +### 2.3 Ensure that the --auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and either remove the --auto-tls parameter or set it to false. + --auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_AUTO_TLS' is not present OR 'ETCD_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-14-226 ETCD_UNSUPPORTED_ARCH= FILE_HASH=e9ca6f328e70dd3c17ba78c302ee32927a4961198e95d1d948114a5d7e350d99 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=aa1658a59ab75324ef67c786d08307e3 HOME=/ +``` + +### 2.4 Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Follow the etcd service documentation and configure peer TLS encryption as appropriate +for your etcd cluster. +Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the +master node and set the below parameters. +--peer-client-file= +--peer-key-file= +Not Applicable. + +### 2.5 Ensure that the --peer-client-cert-auth argument is set to true (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and set the below parameter. +--peer-client-cert-auth=true +Not Applicable. + +### 2.6 Ensure that the --peer-auto-tls argument is not set to true (Automated) + + +**Result:** pass + +**Remediation:** +Edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the master +node and either remove the --peer-auto-tls parameter or set it to false. +--peer-auto-tls=false + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Expected Result**: + +```console +'ETCD_PEER_AUTO_TLS' is not present OR 'ETCD_PEER_AUTO_TLS' is present +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-14-226 ETCD_UNSUPPORTED_ARCH= FILE_HASH=e9ca6f328e70dd3c17ba78c302ee32927a4961198e95d1d948114a5d7e350d99 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=aa1658a59ab75324ef67c786d08307e3 HOME=/ +``` + +### 2.7 Ensure that a unique Certificate Authority is used for etcd (Automated) + + +**Result:** pass + +**Remediation:** +[Manual test] +Follow the etcd documentation and create a dedicated certificate authority setup for the +etcd service. +Then, edit the etcd pod specification file /var/lib/rancher/rke2/agent/pod-manifests/etcd.yaml on the +master node and set the below parameter. +--trusted-ca-file= + +**Audit:** + +```bash +/bin/ps -ef | /bin/grep etcd | /bin/grep -v grep +``` + +**Audit Config:** + +```bash +cat /var/lib/rancher/rke2/server/db/etcd/config +``` + +**Expected Result**: + +```console +'ETCD_TRUSTED_CA_FILE' is present OR '{.peer-transport-security.trusted-ca-file}' is equal to '/var/lib/rancher/rke2/server/tls/etcd/peer-ca.crt' +``` + +**Returned Value**: + +```console +PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOSTNAME=ip-172-31-14-226 ETCD_UNSUPPORTED_ARCH= FILE_HASH=e9ca6f328e70dd3c17ba78c302ee32927a4961198e95d1d948114a5d7e350d99 NO_PROXY=.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16 POD_HASH=aa1658a59ab75324ef67c786d08307e3 HOME=/ +``` + +## 3.1 Authentication and Authorization +### 3.1.1 Client certificate authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be +implemented in place of client certificates. + +### 3.1.2 Service account token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of service account tokens. + +### 3.1.3 Bootstrap token authentication should not be used for users (Manual) + + +**Result:** warn + +**Remediation:** +Alternative mechanisms provided by Kubernetes such as the use of OIDC should be implemented +in place of bootstrap tokens. + +## 3.2 Logging +### 3.2.1 Ensure that a minimal audit policy is created (Automated) + + +**Result:** pass + +**Remediation:** +Create an audit policy file for your cluster. + +**Audit:** + +```bash +/bin/ps -ef | grep kube-apiserver | grep -v grep +``` + +**Expected Result**: + +```console +'--audit-policy-file' is present +``` + +**Returned Value**: + +```console +root 2489 2419 8 Sep11 ? 01:41:54 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --admission-control-config-file=/etc/rancher/rke2/config/rancher-psact.yaml --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key root 2652 2539 2 Sep11 ? 00:24:53 kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cert-dir=/var/lib/rancher/rke2/server/tls/kube-controller-manager --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true +``` + +### 3.2.2 Ensure that the audit policy covers key security concerns (Manual) + + +**Result:** warn + +**Remediation:** +Review the audit policy provided for the cluster and ensure that it covers +at least the following areas, +- Access to Secrets managed by the cluster. Care should be taken to only + log Metadata for requests to Secrets, ConfigMaps, and TokenReviews, in + order to avoid risk of logging sensitive data. +- Modification of Pod and Deployment objects. +- Use of `pods/exec`, `pods/portforward`, `pods/proxy` and `services/proxy`. +For most requests, minimally logging at the Metadata level is recommended +(the most basic level of logging). + +## 4.1 Worker Node Configuration Files +### 4.1.1 Ensure that the kubelet service file permissions are set to 600 or more restrictive (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chmod 600 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not Applicable. + +### 4.1.2 Ensure that the kubelet service file ownership is set to root:root (Automated) + + +**Result:** Not Applicable + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +Not applicable. + +### 4.1.3 If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive (Manual) + + +**Result:** warn + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.4 If proxy kubeconfig file exists ensure ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, chown root:root /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubeproxy.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.5 Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chmod 600 /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.6 Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the below command (based on the file location on your system) on the each worker node. +For example, +chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.7 Ensure that the certificate authorities file permissions are set to 600 or more restrictive (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the file permissions of the +--client-ca-file chmod 600 + +**Audit Script:** `check_cafile_permissions.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +CAFILE=/node$CAFILE +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c permissions=%a $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_permissions.sh +``` + +**Expected Result**: + +```console +permissions has permissions 600, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=600 +``` + +### 4.1.8 Ensure that the client certificate authorities file ownership is set to root:root (Manual) + + +**Result:** pass + +**Remediation:** +Run the following command to modify the ownership of the --client-ca-file. +chown root:root + +**Audit Script:** `check_cafile_ownership.sh` + +```bash +#!/usr/bin/env bash + +CAFILE=$(ps -ef | grep kubelet | grep -v apiserver | grep -- --client-ca-file= | awk -F '--client-ca-file=' '{print $2}' | awk '{print $1}') +CAFILE=/node$CAFILE +if test -z $CAFILE; then CAFILE=$kubeletcafile; fi +if test -e $CAFILE; then stat -c %U:%G $CAFILE; fi + +``` + +**Audit Execution:** + +```bash +./check_cafile_ownership.sh +``` + +**Expected Result**: + +```console +'root:root' is equal to 'root:root' +``` + +**Returned Value**: + +```console +root:root +``` + +### 4.1.9 If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive (Automated) + + +**Result:** fail + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chmod 600 /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c permissions=%a /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +permissions has permissions 644, expected 600 or more restrictive +``` + +**Returned Value**: + +```console +permissions=644 +``` + +### 4.1.10 If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root (Automated) + + +**Result:** pass + +**Remediation:** +Run the following command (using the config file location identified in the Audit step) +chown root:root /var/lib/rancher/rke2/agent/kubelet.kubeconfig + +**Audit:** + +```bash +/bin/sh -c 'if test -e /var/lib/rancher/rke2/agent/kubelet.kubeconfig; then stat -c %U:%G /var/lib/rancher/rke2/agent/kubelet.kubeconfig; fi' +``` + +**Expected Result**: + +```console +'root:root' is present +``` + +**Returned Value**: + +```console +root:root +``` + +## 4.2 Kubelet +### 4.2.1 Ensure that the --anonymous-auth argument is set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication: anonymous: enabled` to +`false`. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +`--anonymous-auth=false` +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--anonymous-auth' is equal to 'false' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.2 Ensure that the --authorization-mode argument is not set to AlwaysAllow (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authorization.mode` to Webhook. If +using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--authorization-mode=Webhook +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--authorization-mode' does not have 'AlwaysAllow' +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.3 Ensure that the --client-ca-file argument is set as appropriate (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `authentication.x509.clientCAFile` to +the location of the client CA file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_AUTHZ_ARGS variable. +--client-ca-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--client-ca-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.4 Verify that the --read-only-port argument is set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `readOnlyPort` to 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--read-only-port=0 +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--read-only-port' is equal to '0' OR '--read-only-port' is not present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.5 Ensure that the --streaming-connection-idle-timeout argument is not set to 0 (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `streamingConnectionIdleTimeout` to a +value other than 0. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +--streaming-connection-idle-timeout=5m +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.streamingConnectionIdleTimeout}' is present OR '{.streamingConnectionIdleTimeout}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.6 Ensure that the --make-iptables-util-chains argument is set to true (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `makeIPTablesUtilChains` to `true`. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove the --make-iptables-util-chains argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.makeIPTablesUtilChains}' is present OR '{.makeIPTablesUtilChains}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.7 Ensure that the --hostname-override argument is not set (Manual) + + +**Result:** Not Applicable + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and remove the --hostname-override argument from the +KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service +Not Applicable. + +### 4.2.8 Ensure that the eventRecordQPS argument is set to a level which ensures appropriate event capture (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `eventRecordQPS` to an appropriate level. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameter in KUBELET_SYSTEM_PODS_ARGS variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.eventRecordQPS}' is present OR '{.eventRecordQPS}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.9 Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to set `tlsCertFile` to the location +of the certificate file to use to identify this Kubelet, and `tlsPrivateKeyFile` +to the location of the corresponding private key file. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the below parameters in KUBELET_CERTIFICATE_ARGS variable. +--tls-cert-file= +--tls-private-key-file= +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'--tls-cert-file' is present AND '--tls-private-key-file' is present +``` + +**Returned Value**: + +```console +UID PID PPID C STIME TTY TIME CMD root 2236 2075 4 Sep11 ? 00:47:51 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=ip-172-31-14-226 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-labels=cattle.io/os=linux,rke.cattle.io/machine=1adcc1f7-66f1-4503-b9bc-dfb6e808f27b --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --protect-kernel-defaults=true --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key +``` + +### 4.2.10 Ensure that the --rotate-certificates argument is not set to false (Automated) + + +**Result:** pass + +**Remediation:** +If using a Kubelet config file, edit the file to add the line `rotateCertificates` to `true` or +remove it altogether to use the default value. +If using command line arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +remove --rotate-certificates=false argument from the KUBELET_CERTIFICATE_ARGS +variable. +Based on your system, restart the kubelet service. For example, +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.rotateCertificates}' is present OR '{.rotateCertificates}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.11 Verify that the RotateKubeletServerCertificate argument is set to true (Manual) + + +**Result:** pass + +**Remediation:** +Edit the kubelet service file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf +on each worker node and set the below parameter in KUBELET_CERTIFICATE_ARGS variable. +--feature-gates=RotateKubeletServerCertificate=true +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.featureGates.RotateKubeletServerCertificate}' is present OR '{.featureGates.RotateKubeletServerCertificate}' is not present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.12 Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers (Manual) + + +**Result:** warn + +**Remediation:** +If using a Kubelet config file, edit the file to set `TLSCipherSuites` to +TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +or to a subset of these values. +If using executable arguments, edit the kubelet service file +/etc/systemd/system/kubelet.service.d/10-kubeadm.conf on each worker node and +set the --tls-cipher-suites parameter as follows, or to a subset of these values. +--tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_GCM_SHA256 +Based on your system, restart the kubelet service. For example: +systemctl daemon-reload +systemctl restart kubelet.service + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{range .tlsCipherSuites[:]}{}{','}{end}' is present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +### 4.2.13 Ensure that a limit is set on pod PIDs (Manual) + + +**Result:** warn + +**Remediation:** +Decide on an appropriate level for this parameter and set it, +either via the --pod-max-pids command line parameter or the PodPidsLimit configuration file setting. + +**Audit:** + +```bash +/bin/ps -fC kubelet +``` + +**Audit Config:** + +```bash +/bin/cat /var/lib/rancher/rke2/agent/kubelet.kubeconfig +``` + +**Expected Result**: + +```console +'{.podPidsLimit}' is present +``` + +**Returned Value**: + +```console +apiVersion: v1 clusters: - cluster: server: https://127.0.0.1:6443 certificate-authority: /var/lib/rancher/rke2/agent/server-ca.crt name: local contexts: - context: cluster: local namespace: default user: user name: Default current-context: Default kind: Config preferences: {} users: - name: user user: client-certificate: /var/lib/rancher/rke2/agent/client-kubelet.crt client-key: /var/lib/rancher/rke2/agent/client-kubelet.key +``` + +## 5.1 RBAC and Service Accounts +### 5.1.1 Ensure that the cluster-admin role is only used where required (Manual) + + +**Result:** warn + +**Remediation:** +Identify all clusterrolebindings to the cluster-admin role. Check if they are used and +if they need this role or if they could use a role with fewer privileges. +Where possible, first bind users to a lower privileged role and then remove the +clusterrolebinding to the cluster-admin role : +kubectl delete clusterrolebinding [name] + +### 5.1.2 Minimize access to secrets (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove get, list and watch access to Secret objects in the cluster. + +### 5.1.3 Minimize wildcard use in Roles and ClusterRoles (Manual) + + +**Result:** warn + +**Remediation:** +Where possible replace any use of wildcards in clusterroles and roles with specific +objects or actions. + +### 5.1.4 Minimize access to create pods (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to pod objects in the cluster. + +### 5.1.5 Ensure that default service accounts are not actively used. (Manual) + + +**Result:** warn + +**Remediation:** +Create explicit service accounts wherever a Kubernetes workload requires specific access +to the Kubernetes API server. +Modify the configuration of each default service account to include this value +automountServiceAccountToken: false + +### 5.1.6 Ensure that Service Account Tokens are only mounted where necessary (Manual) + + +**Result:** warn + +**Remediation:** +Modify the definition of pods and service accounts which do not need to mount service +account tokens to disable it. + +### 5.1.7 Avoid use of system:masters group (Manual) + + +**Result:** warn + +**Remediation:** +Remove the system:masters group from all users in the cluster. + +### 5.1.8 Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove the impersonate, bind and escalate rights from subjects. + +### 5.1.9 Minimize access to create persistent volumes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove create access to PersistentVolume objects in the cluster. + +### 5.1.10 Minimize access to the proxy sub-resource of nodes (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the proxy sub-resource of node objects. + +### 5.1.11 Minimize access to the approval sub-resource of certificatesigningrequests objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the approval sub-resource of certificatesigningrequest objects. + +### 5.1.12 Minimize access to webhook configuration objects (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the validatingwebhookconfigurations or mutatingwebhookconfigurations objects + +### 5.1.13 Minimize access to the service account token creation (Manual) + + +**Result:** warn + +**Remediation:** +Where possible, remove access to the token sub-resource of serviceaccount objects. + +## 5.2 Pod Security Standards +### 5.2.1 Ensure that the cluster has at least one active policy control mechanism in place (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that either Pod Security Admission or an external policy control system is in place +for every namespace which contains user workloads. + +### 5.2.2 Minimize the admission of privileged containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of privileged containers. + +### 5.2.3 Minimize the admission of containers wishing to share the host process ID namespace (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostPID` containers. + +### 5.2.4 Minimize the admission of containers wishing to share the host IPC namespace (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostIPC` containers. + +### 5.2.5 Minimize the admission of containers wishing to share the host network namespace (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of `hostNetwork` containers. + +### 5.2.6 Minimize the admission of containers with allowPrivilegeEscalation (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `.spec.allowPrivilegeEscalation` set to `true`. + +### 5.2.7 Minimize the admission of root containers (Manual) + + +**Result:** warn + +**Remediation:** +Create a policy for each namespace in the cluster, ensuring that either `MustRunAsNonRoot` +or `MustRunAs` with the range of UIDs not including 0, is set. + +### 5.2.8 Minimize the admission of containers with the NET_RAW capability (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with the `NET_RAW` capability. + +### 5.2.9 Minimize the admission of containers with added capabilities (Automated) + + +**Result:** warn + +**Remediation:** +Ensure that `allowedCapabilities` is not present in policies for the cluster unless +it is set to an empty array. + +### 5.2.10 Minimize the admission of containers with capabilities assigned (Manual) + + +**Result:** warn + +**Remediation:** +Review the use of capabilites in applications running on your cluster. Where a namespace +contains applicaions which do not require any Linux capabities to operate consider adding +a PSP which forbids the admission of containers which do not drop all capabilities. + +### 5.2.11 Minimize the admission of Windows HostProcess containers (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers that have `.securityContext.windowsOptions.hostProcess` set to `true`. + +### 5.2.12 Minimize the admission of HostPath volumes (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers with `hostPath` volumes. + +### 5.2.13 Minimize the admission of containers which use HostPorts (Manual) + + +**Result:** warn + +**Remediation:** +Add policies to each namespace in the cluster which has user workloads to restrict the +admission of containers which use `hostPort` sections. + +## 5.3 Network Policies and CNI +### 5.3.1 Ensure that the CNI in use supports NetworkPolicies (Manual) + + +**Result:** warn + +**Remediation:** +If the CNI plugin in use does not support network policies, consideration should be given to +making use of a different plugin, or finding an alternate mechanism for restricting traffic +in the Kubernetes cluster. + +### 5.3.2 Ensure that all Namespaces have NetworkPolicies defined (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create NetworkPolicy objects as you need them. + +## 5.4 Secrets Management +### 5.4.1 Prefer using Secrets as files over Secrets as environment variables (Manual) + + +**Result:** warn + +**Remediation:** +If possible, rewrite application code to read Secrets from mounted secret files, rather than +from environment variables. + +### 5.4.2 Consider external secret storage (Manual) + + +**Result:** warn + +**Remediation:** +Refer to the Secrets management options offered by your cloud provider or a third-party +secrets management solution. + +## 5.5 Extensible Admission Control +### 5.5.1 Configure Image Provenance using ImagePolicyWebhook admission controller (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and setup image provenance. + +## 5.7 General Policies +### 5.7.1 Create administrative boundaries between resources using namespaces (Manual) + + +**Result:** warn + +**Remediation:** +Follow the documentation and create namespaces for objects in your deployment as you need +them. + +### 5.7.2 Ensure that the seccomp profile is set to docker/default in your Pod definitions (Manual) + + +**Result:** warn + +**Remediation:** +Use `securityContext` to enable the docker/default seccomp profile in your pod definitions. +An example is as below: + securityContext: + seccompProfile: + type: RuntimeDefault + +### 5.7.3 Apply SecurityContext to your Pods and Containers (Manual) + + +**Result:** warn + +**Remediation:** +Follow the Kubernetes documentation and apply SecurityContexts to your Pods. For a +suggested list of SecurityContexts, you may refer to the CIS Security Benchmark for Docker +Containers. + +### 5.7.4 The default namespace should not be used (Manual) + + +**Result:** warn + +**Remediation:** +Ensure that namespaces are created to allow for appropriate segregation of Kubernetes +resources and that all new resources are created in a specific namespace. + diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/rancher-security-best-practices.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/rancher-security-best-practices.md new file mode 100644 index 00000000000..0d98495ffec --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/rancher-security-best-practices.md @@ -0,0 +1,21 @@ +--- +title: Rancher 安全最佳实践 +--- + + + + + +### 限制对 /version 和 /rancherversion 的公共访问 + +上游（本地） Rancher 实例提供正在运行的 Rancher 版本和用于构建它的 Go 版本信息。这些信息可以通过 `/version` 路径访问，该路径用于诸如自动化版本升级或确认部署成功等任务。上游实例还提供了可通过 `/rancherversion` 路径访问的 Rancher 版本信息。 + +攻击者可能会滥用这些信息来识别正在运行的 Rancher 版本，并与潜在的漏洞相关联以进行利用。如果你的上游 Rancher 实例在网上是公开可访问的，请使用 7 层防火墙来阻止 `/version` 和 `/rancherversion` 路径。 + +更多关于保护服务器的详细信息，请参阅 [OWASP Web Application Security Testing - Enumerate Infrastructure and Application Admin Interfaces](https://owasp.org/www-project-web-security-testing-guide/stable/4-Web_Application_Security_Testing/02-Configuration_and_Deployment_Management_Testing/05-Enumerate_Infrastructure_and_Application_Admin_Interfaces.html)。 + +### 会话管理 + +某些环境可能需要额外的安全控制来管理会话。例如，你可能希望限制用户的并发活动会话或限制可以从哪些地理位置发起这些会话。Rancher 默认情况下不支持这些功能。 + +如果你需要此类功能，请将 7 层防火墙与[外部认证](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/authentication-config/authentication-config.md#外部认证与本地认证)结合使用。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/rancher-security.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/rancher-security.md new file mode 100644 index 00000000000..935aaa2b780 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/rancher-security.md @@ -0,0 +1,95 @@ +--- +title: 安全 +--- + + + + + + + + + + + +
+

安全策略

+

Rancher Labs 会负责任地披露问题，并致力于在合理的时间内解决所有问题。

+		 +

报告流程

+

请将安全问题发送至 security-rancher@suse.com。

+		 +

公告

+

订阅 Rancher 公告论坛以获取版本更新。

+
+ +安全是 Rancher 全部功能的基础。Rancher 集成了全部主流认证工具和服务，并提供了企业级的 [RBAC 功能](../../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/manage-role-based-access-control-rbac.md)，让你的 Kubernetes 集群更加安全。 + +本文介绍了安全相关的文档以及资源，让你的 Rancher 安装和下游 Kubernetes 集群更加安全。 + +### NeuVector 与 Rancher 的集成 + +NeuVector 是一个开源的、以容器为中心的安全应用程序，现已集成到 Rancher 中。NeuVector 提供生产安全、DevOps 漏洞保护和容器防火墙等功能。请参阅 [Rancher 文档](../../integrations-in-rancher/neuvector/neuvector.md) 和 [NeuVector 文档](https://open-docs.neuvector.com/)了解更多信息。 + +### 在 Kubernetes 集群上运行 CIS 安全扫描 + +Rancher 使用 [kube-bench](https://github.com/aquasecurity/kube-bench) 来运行安全扫描，从而检查 Kubernetes 是否按照 [CIS](https://www.cisecurity.org/cis-benchmarks/)（Center for Internet Security，互联网安全中心）Kubernetes Benchmark 中定义的安全最佳实践进行部署。 + +CIS Kubernetes Benchmark 是一个参考文档，用于为 Kubernetes 建立安全配置基线。 + +CIS 是一个 501(c\)(3) 非营利组织，成立于 2000 年 10 月，其使命是识别、开发、验证、促进和维持网络防御的最佳实践方案，并建立和指导社区，以在网络空间中营造信任的环境。 + +CIS Benchmark 是目标系统安全配置的最佳实践。CIS Benchmark 是由安全专家、技术供应商、公开和私人社区成员，以及 CIS Benchmark 开发团队共同志愿开发的。 + +Benchmark 提供两种类型的建议，分别是自动（Automated）和手动（Manual）。我们只运行 Automated 相关的测试。 + +Rancher 在集群上运行 CIS 安全扫描时会生成一份报告，该报告会显示每个测试的结果，包括测试概要以及 `passed`、`skipped` 和 `failed` 的测试数量。报告还包括失败测试的修正步骤。 + +有关详细信息，请参阅[安全扫描](../../how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md)。 + +### SELinux RPM + +[安全增强型 Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) 是对 Linux 的安全增强。被政府机构使用之后，SELinux 已成为行业标准，并在 CentOS 7 和 8 上默认启用。 + +我们提供了 `rancher-selinux` 和 `rke2-selinux` 两个 RPM（Red Hat 软件包），让 Rancher 产品能够在 SELinux 主机上正常运行。有关详细信息，请参阅[此页面](selinux-rpm/selinux-rpm.md)。 + +### Rancher 加固指南 + +Rancher 加固指南基于 CIS Kubernetes Benchmark。 + +加固指南为加固 Rancher 的生产安装提供了说明性指导。有关安全管控的完整列表，请参阅 Rancher 的 [CIS Kubernetes Benchmark 自我评估](#cis-benchmark-和自我评估)指南。 + +> 加固指南描述了如何保护集群中的节点，建议在安装 Kubernetes 之前参考加固指南中的步骤。 + +每个加固指南版本都针对特定的 CIS Kubernetes Benchmark、Kubernetes 和 Rancher 版本。 + +### CIS Benchmark 和自我评估 + +Benchmark 自我评估是 Rancher 安全加固指南的辅助。加固指南展示了如何加固集群，而 Benchmark 指南旨在帮助你评估加固集群的安全级别。 + +由于 Rancher 和 RKE 将 Kubernetes 服务安装为 Docker 容器，因此 CIS Kubernetes Benchmark 中的许多管控验证检查都不适用。本指南将介绍各种 controls，并提供更新的示例命令来审计 Rancher 创建的集群的合规性。你可以前往 [CIS 网站](https://www.cisecurity.org/benchmark/kubernetes/)下载原始的 Benchmark 文档。 + +Rancher 自我评估指南的每个版本都对应于强化指南、Rancher、Kubernetes 和 CIS Benchmark 的特定版本。 + +### 第三方渗透测试报告 + +Rancher 会定期聘请第三方对 Rancher 软件栈进行安全审计和渗透测试。被测环境遵循 Rancher 在测试时提供的强化指南。以前的渗透测试报告如下。 + +结果： + +- [Cure53 渗透测试 - 2019 年 7 月](https://releases.rancher.com/documents/security/pen-tests/2019/RAN-01-cure53-report.final.pdf) +- [Untamed Theory 渗透测试 - 2019 年 3 月](https://releases.rancher.com/documents/security/pen-tests/2019/UntamedTheory-Rancher_SecurityAssessment-20190712_v5.pdf) + +请注意，新报告不再共享或公开发布。 + +### Rancher 安全公告和 CVE + +Rancher 致力于向社区通报我们产品中的安全问题。有关我们已解决的问题的 CVE（常见漏洞和暴露）列表，请参阅[此页](security-advisories-and-cves.md)。 + +### Kubernetes 安全最佳实践 + +有关保护 Kubernetes 集群的建议，请参阅 [Kubernetes 安全最佳实践](kubernetes-security-best-practices.md)指南。 + +### Rancher 安全最佳实践 + +有关保护 Rancher Manager 部署的建议，请参阅 [Rancher 安全最佳实践](rancher-security-best-practices.md)指南。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/security-advisories-and-cves.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/security-advisories-and-cves.md index b34a5f15626..c39522aaa1b 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/security-advisories-and-cves.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/security-advisories-and-cves.md @@ -2,10 +2,18 @@ title: 安全公告和 CVE --- + + + + Rancher 致力于向社区披露我们产品的安全问题。我们会针对已解决的问题发布安全公告和 CVE（Common Vulnerabilities and Exposures，通用漏洞披露）。Rancher GitHub 上的[安全页面](https://github.com/rancher/rancher/security/advisories)也会发布新的安全公告。 | ID | 描述 | 日期 | 解决 | |----|-------------|------|------------| +| [CVE-2023-32193](https://github.com/rancher/norman/security/advisories/GHSA-r8f4-hv23-6qp6) | 在 Rancher 2.6.13、2.7.9 和 2.8.1 及之前的版本中发现了一个问题。多个 Cross-Site Scripting (XSS) 漏洞可通过 Rancher UI (Norman) 进行利用。 | 2024 年 2 月 8 日 | Rancher [v2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2)、[v2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) 和 [v2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | +| [CVE-2023-32192](https://github.com/rancher/apiserver/security/advisories/GHSA-833m-37f7-jq55) | 在 Rancher 2.6.13、2.7.9 和 2.8.1 及之前的版本中发现了一个问题。多个 Cross-Site Scripting (XSS) 漏洞，可以通过 Rancher UI (Apiserver) 进行利用 | 2024 年 2 月 8 日 | Rancher [v2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2)、[v2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) 和 [v2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | +| [CVE-2023-22649](https://github.com/rancher/rancher/security/advisories/GHSA-xfj7-qf8w-2gcr) | 在 Rancher 2.6.13、2.7.9 和 2.8.1 及之前的版本中发现了一个问题。敏感数据可能会泄漏到 Rancher 的审计日志中。 | 2024 年 2 月 8 日 | Rancher [v2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2)、[v2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) 和 [v2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | +| [CVE-2023-32194](https://github.com/rancher/rancher/security/advisories/GHSA-c85r-fwc7-45vc) | 在 Rancher 2.6.13、2.7.9 和 2.8.1 及之前的版本中发现了一个问题。当为 “namespace” 资源类型授予 `create` 或 `*` 全局角色时，任何 API 组中拥有权限的用户可以管理核心 API 组中的 namespace。 | 2024 年 2 月 8 日 | Rancher [v2.8.2](https://github.com/rancher/rancher/releases/tag/v2.8.2)、[v2.7.10](https://github.com/rancher/rancher/releases/tag/v2.7.10) 和 [v2.6.14](https://github.com/rancher/rancher/releases/tag/v2.6.14) | | [CVE-2023-22648](https://github.com/rancher/rancher/security/advisories/GHSA-vf6j-6739-78m8) | 在 Rancher 2.6.12 和 2.7.3 及之前的版本中发现了一个问题。在用户注销并重新登录到 Rancher UI 之前，Azure AD 中的权限更改不会反映给用户。 | 2023 年 5 月 31 日 | Rancher [v2.7.4](https://github.com/rancher/rancher/releases/tag/v2.7.4) | | [CVE-2022-43760](https://github.com/rancher/rancher/security/advisories/GHSA-46v3-ggjg-qq3x) | 在 Rancher 2.6.12 和 2.7.3 及之前的版本中发现了一个问题。攻击者可以通过 Rancher UI 利用多个跨站脚本 (XSS) 漏洞。 | 2023 年 5 月 31 日 | Rancher [v2.7.4](https://github.com/rancher/rancher/releases/tag/v2.7.4) | | [CVE-2020-10676](https://github.com/rancher/rancher/security/advisories/GHSA-8vhc-hwhc-cpj4) | 在 Rancher 2.6.12 和 2.7.3 及之前的版本中发现了一个问题。具有更新命名空间权限的用户可以将该命名空间移动到他们无权访问的项目中。 | 2023 年 5 月 31 日 | Rancher [v2.7.4](https://github.com/rancher/rancher/releases/tag/v2.7.4) | @@ -19,10 +27,10 @@ Rancher 致力于向社区披露我们产品的安全问题。我们会针对已 | [CVE-2022-31247](https://github.com/rancher/rancher/security/advisories/GHSA-6x34-89p7-95wg) | 在 Rancher 2.5.15 和 2.6.6 及之前的版本中发现了一个问题。授权逻辑缺陷允许在下游集群中通过集群角色模板绑定 (CRTB) 和项目角色模板绑定 (PRTB) 来提升权限。任何有权限创建/编辑 CRTB 或 PRTB 的用户（例如 `cluster-owner`、`manage cluster members`、`project-owner` 和 `manage project members`）都可以利用该漏洞，在同一集群的另一个项目或不同下游集群的另一个项目中获得所有者权限。 | 2022 年 8 月 18 日 | [Rancher 2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) 和 [Rancher 2.5.16](https://github.com/rancher/rancher/releases/tag/v2.5.16) | | [CVE-2021-36783](https://github.com/rancher/rancher/security/advisories/GHSA-8w87-58w6-hfv8) | 2.5.12 到 2.6.3 的 Rancher 版本无法正确清理集群模板 answer 中的凭证。此错误可能会导致明文存储以及凭证、密码和 API 令牌被暴露。在 Rancher 中，已认证的 `Cluster Owner`、`Cluster Member`、`Project Owner` 和 `Project Member` 可以在 `/v1/management.cattle.io.clusters`、`/v3/clusters` 和 `/k8s/clusters/local/apis/management.cattle.io/v3/clusters` 端点上看到暴露的凭证。 | 2022 年 8 月 18 日 | [Rancher 2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) 和 [Rancher 2.5.16](https://github.com/rancher/rancher/releases/tag/v2.5.16) | | [CVE-2021-36782](https://github.com/rancher/rancher/security/advisories/GHSA-g7j7-h4q8-8w2f) | 在 2.5.15 到 2.6.6 的 Rancher 版本中发现了一个问题，其中密码、API 密钥和 Rancher 的 ServiceAccount 令牌（用于配置集群）等敏感字段直接以明文形式存储在 `Cluster` 等 Kubernetes 对象上（例如，`cluster.management.cattle.io`）。任何能够读取 Kubernetes API 中的对象的用户都可以检索这些敏感数据的明文版本。该问题由 Florian Struck（来自 [Continum AG](https://www.continum.net/)）和 [Marco Stuurman](https://github.com/fe-ax)（来自 [Shock Media B.V.](https://www.shockmedia.nl/)）发现并报告。 | 2022 年 8 月 18 日 | [Rancher 2.6.7](https://github.com/rancher/rancher/releases/tag/v2.6.7) 和 [Rancher 2.5.16](https://github.com/rancher/rancher/releases/tag/v2.5.16) | -| [CVE-2022-21951](https://github.com/rancher/rancher/security/advisories/GHSA-vrph-m5jj-c46c) | 此漏洞仅影响通过 [RKE 模板](https://rancher.com/docs/rancher/v2.6/en/admin-settings/rke-templates/)配置 [Weave](https://rancher.com/docs/rancher/v2.6/en/faq/networking/cni-providers/#weave) 容器网络接口 (CNI) 的客户。在 Rancher 2.5.0 到 2.5.13 和 Rancher 2.6.0 到 2.6.4 版本中发现了一个漏洞。如果将 CNI 选为 Weave，RKE 模板的用户界面 (UI) 不包括 Weave 密码的值。如果基于上述模板创建集群，并且将 Weave 配置为 CNI，则 Weave 中不会为[网络加密](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/)创建密码。因此，集群中的网络流量将不加密发送。 | 2022 年 5 月 24 日 | [Rancher 2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) 和 [Rancher 2.5.14](https://github.com/rancher/rancher/releases/tag/v2.5.14) | -| [CVE-2021-36784](https://github.com/rancher/rancher/security/advisories/GHSA-jwvr-vv7p-gpwq) | 在 Rancher 2.5.0 到 2.5.12 和 Rancher 2.6.0 到 2.6.3 中发现了一个漏洞，该漏洞允许能创建或更新[全局角色](https://rancher.com/docs/rancher/v2.6/en/admin-settings/rbac/)的用户将他们或其他用户升级为管理员。全局角色能授予用户 Rancher 级别的权限，例如能创建集群。在已识别的 Rancher 版本中，如果用户被授予了编辑或创建全局角色的权限，他们不仅仅能授予他们已经拥有的权限。此漏洞影响使用能够创建或编辑全局角色的非管理员用户的客户。此场景最常见的用例是 `restricted-admin` 角色。 | 2022 年 4 月 14 日 | [Rancher 2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) 和 [Rancher 2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | +| [CVE-2022-21951](https://github.com/rancher/rancher/security/advisories/GHSA-vrph-m5jj-c46c) | 此漏洞仅影响通过 [RKE 模板](../../pages-for-subheaders/about-rke1-templates.md)配置 [Weave](../../faq/container-network-interface-providers.md#weave) 容器网络接口 (CNI) 的客户。在 Rancher 2.5.0 到 2.5.13 和 Rancher 2.6.0 到 2.6.4 版本中发现了一个漏洞。如果将 CNI 选为 Weave，RKE 模板的用户界面 (UI) 不包括 Weave 密码的值。如果基于上述模板创建集群，并且将 Weave 配置为 CNI，则 Weave 中不会为[网络加密](https://www.weave.works/docs/net/latest/tasks/manage/security-untrusted-networks/)创建密码。因此，集群中的网络流量将不加密发送。 | 2022 年 5 月 24 日 | [Rancher 2.6.5](https://github.com/rancher/rancher/releases/tag/v2.6.5) 和 [Rancher 2.5.14](https://github.com/rancher/rancher/releases/tag/v2.5.14) | +| [CVE-2021-36784](https://github.com/rancher/rancher/security/advisories/GHSA-jwvr-vv7p-gpwq) | 在 Rancher 2.5.0 到 2.5.12 和 Rancher 2.6.0 到 2.6.3 中发现了一个漏洞，该漏洞允许能创建或更新[全局角色](../../pages-for-subheaders/manage-role-based-access-control-rbac.md)的用户将他们或其他用户升级为管理员。全局角色能授予用户 Rancher 级别的权限，例如能创建集群。在已识别的 Rancher 版本中，如果用户被授予了编辑或创建全局角色的权限，他们不仅仅能授予他们已经拥有的权限。此漏洞影响使用能够创建或编辑全局角色的非管理员用户的客户。此场景最常见的用例是 `restricted-admin` 角色。 | 2022 年 4 月 14 日 | [Rancher 2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) 和 [Rancher 2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | | [CVE-2021-4200](https://github.com/rancher/rancher/security/advisories/GHSA-hx8w-ghh8-r4xf) | 此漏洞仅影响在 Rancher 中使用 `restricted-admin` 角色的客户。在 Rancher 2.5.0 到 2.5.12 和 2.6.0 到 2.6.3 中发现了一个漏洞，其中 `cattle-global-data` 命名空间中的 `global-data` 角色授予了应用商店的写权限。由于具有任何级别的应用商店访问权限的用户都会绑定到 `global-data` 角色，因此这些用户都能写入模板 `CatalogTemplates`) 和模板版本 (`CatalogTemplateVersions`)。在 Rancher 中创建的新用户默认分配到 `user` 角色（普通用户），该角色本不该具有写入应用商店的权限。此漏洞提升了能写入应用商店模板和应用商店模板版本资源的用户的权限。 | 2022 年 4 月 14 日 | [Rancher 2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) 和 [Rancher 2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | -| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | 此漏洞仅影响使用经过认证的 Git 和/或 Helm 仓库通过 [Fleet](https://rancher.com/docs/rancher/v2.6/en/deploy-across-clusters/fleet/) 进行持续交付的客户。在 [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) 之前版本中的 `go-getter` 库中发现了一个问题，错误消息中没有删除 Base64 编码的 SSH 私钥，导致该信息暴露。Rancher 中 [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9) 之前的 Fleet 版本使用了该库的漏洞版本。此问题影响 Rancher 2.5.0 到 2.5.12（包括 2.5.12）以及 2.6.0 到 2.6.3（包括 2.6.3）。该问题由 Raft Engineering 的 Dagan Henderson 发现并报告。 | 2022 年 4 月 14 日 | [Rancher 2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) 和 [Rancher 2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | +| [GHSA-wm2r-rp98-8pmh](https://github.com/rancher/rancher/security/advisories/GHSA-wm2r-rp98-8pmh) | 此漏洞仅影响使用经过认证的 Git 和/或 Helm 仓库通过 [Fleet](../../how-to-guides/new-user-guides/deploy-apps-across-clusters/fleet.md) 进行持续交付的客户。在 [`v1.5.11`](https://github.com/hashicorp/go-getter/releases/tag/v1.5.11) 之前版本中的 `go-getter` 库中发现了一个问题，错误消息中没有删除 Base64 编码的 SSH 私钥，导致该信息暴露。Rancher 中 [`v0.3.9`](https://github.com/rancher/fleet/releases/tag/v0.3.9) 之前的 Fleet 版本使用了该库的漏洞版本。此问题影响 Rancher 2.5.0 到 2.5.12（包括 2.5.12）以及 2.6.0 到 2.6.3（包括 2.6.3）。该问题由 Raft Engineering 的 Dagan Henderson 发现并报告。 | 2022 年 4 月 14 日 | [Rancher 2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) 和 [Rancher 2.5.13](https://github.com/rancher/rancher/releases/tag/v2.5.13) | | [CVE-2021-36778](https://github.com/rancher/rancher/security/advisories/GHSA-4fc7-hc63-7fjg) | 在 Rancher 2.5.0 到 2.5.11 和 Rancher 2.6.0 到 2.6.2 中发现了一个漏洞，当从配置的私有仓库下载 Helm Chart 时，对同源策略的检查不足可能导致仓库凭证暴露给第三方提供商。仅当用户在 Rancher 的`应用 & 应用市场 > 仓库`中配置私有仓库的访问凭证时才会出现此问题。该问题由 Martin Andreas Ullrich 发现并报告。 | 2022 年 4 月 14 日 | [Rancher 2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3) 和 [Rancher 2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) | | [GHSA-hwm2-4ph6-w6m5](https://github.com/rancher/rancher/security/advisories/GHSA-hwm2-4ph6-w6m5) | 在 Rancher 2.0 到 2.6.3 中发现了一个漏洞。Rancher 提供的 `restricted` Pod 安全策略（PSP）与 Kubernetes 提供的上游 `restricted` 策略有差别，因此 Rancher 的 PSP 将 `runAsUser` 设置为 `runAsAny`，而上游将 `runAsUser` 设置为 `MustRunAsNonRoot`。因此，即使 Rancher 的 `restricted` 策略是在项目或集群级别上强制执行的，容器也可以以任何用户身份运行，包括特权用户 (`root`)。 | 2022 年 3 月 31 日 | [Rancher 2.6.4](https://github.com/rancher/rancher/releases/tag/v2.6.4) | | [CVE-2021-36775](https://github.com/rancher/rancher/security/advisories/GHSA-28g7-896h-695v) | 在 Rancher 2.4.17、2.5.11 和 2.6.2 以及更高的版本中发现了一个漏洞。从项目中删除与某个组关联的`项目角色`后，能让这些使用者访问集群级别资源的绑定（Binding）不会被删除。导致问题的原因是不完整的授权逻辑检查。如果用户是受影响组中的成员，且能对 Rancher 进行认证访问，那么用户可以利用此漏洞访问他们不应该能访问的资源。暴露级别取决于受影响项目角色的原始权限级别。此漏洞仅影响在 Rancher 中基于组进行身份验证的客户。 | 2022 年 3 月 31 日 | [Rancher 2.6.3](https://github.com/rancher/rancher/releases/tag/v2.6.3)、[Rancher 2.5.12](https://github.com/rancher/rancher/releases/tag/v2.5.12) 和 [Rancher 2.4.18](https://github.com/rancher/rancher/releases/tag/v2.4.18) | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/selinux-rpm/selinux-rpm.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/selinux-rpm/selinux-rpm.md new file mode 100644 index 00000000000..2c96bf912ee --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-security/selinux-rpm/selinux-rpm.md @@ -0,0 +1,20 @@ +--- +title: SELinux RPM +--- + + + + + +[安全增强型 Linux (SELinux)](https://en.wikipedia.org/wiki/Security-Enhanced_Linux) 是对 Linux 的安全增强。 + +它由 Red Hat 开发，是 Linux 上 MAC（mandatory access controls，强制访问控制）的实现。系统管理员可以使用 MAC 设置应用程序和用户是如何访问不同资源的，例如文件、设备、网络和进程间的通信。SELinux 还通过默认限制操作系统来增强安全性。 + +被政府机构使用之后，SELinux 已成为行业标准，并在 CentOS 7 和 8 上默认启用。要检查 SELinux 是否在你的系统上启用和执行，请使用 `getenforce`： + +``` +# getenforce +Enforcing +``` + +我们提供了 [`rancher-selinux`](about-rancher-selinux.md) 和 [`rke2-selinux`](about-rke2-selinux.md) 两个 RPM（Red Hat 软件包），让 Rancher 产品能够在 SELinux 主机上正常运行。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-webhook.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-webhook.md index caa0335ca2a..ee0852497c6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-webhook.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/rancher-webhook.md @@ -2,10 +2,37 @@ title: Rancher Webhook --- + + + + Rancher-Webhook 是 Rancher 的重要组件，它与 Kubernetes 结合使用，用于增强安全性并为 Rancher 管理的集群启用关键功能。 如 [Kubernetes 文档](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/)中所述，它与 Kubernetes 的可扩展准入控制器集成，允许 Rancher-Webhook 检查发送到 Kubernetes API Server 的特定请求，添加自定义和 Rancher 相关的验证，以及 Rancher 相关请求的变化。Rancher-Webhook 使用 `rancher.cattle.io` `ValidatingWebhookConfiguration` 和 `rancher.cattle.io` `MutatingWebhookConfiguration` 管理要验证的资源，并覆盖任何手动编辑。 -Rancher 将 Rancher-Webhook 作为单独的 deployment 和服务部署在 local 和下游集群中。Rancher 使用 Helm 管理 Rancher-Webhook。需要注意的是，Rancher 可能会覆盖用户对 Helm 版本所做的修改。 + +Rancher 将 Rancher-Webhook 作为单独的 deployment 和服务部署在 local 和下游集群中。Rancher 使用 Helm 管理 Rancher-Webhook。需要注意的是，Rancher 可能会覆盖用户对 Helm 版本所做的修改。要安全地修改这些值，请参阅[自定义 Rancher-Webhook 配置](#自定义-rancher-webhook-配置) + +每个 Rancher 版本都设计为与某个具体版本的 Webhook 兼容，为方便起见，下面提供了各版本的兼容列表。 + +**注意：** Rancher 负责管理 webhook 的部署和升级。在多数情况下，不需要用户干预来确保 webhook 版本与你正在运行的 Rancher 版本兼容。 + + + +| Rancher Version | Webhook Version | Availability in Prime | Availability in Community | +| --------------- | --------------- | --------------------- | ------------------------- | +| v2.7.12 | v0.3.7 | ✓ | N/A | +| v2.7.11 | v0.3.7 | ✓ | N/A | +| v2.7.10 | v0.3.6 | ✓ | ✓ | +| v2.7.9 | v0.3.6 | ✗ | ✓ | +| v2.7.8 | v0.3.6 | ✗ | ✓ | +| v2.7.7 | v0.3.6 | ✓ | ✓ | +| v2.7.6 | v0.3.5 | ✓ | ✓ | +| v2.7.5 | v0.3.5 | ✓ | ✓ | +| v2.7.4 | v0.3.4 | ✓ | ✓ | +| v2.7.3 | v0.3.3 | ✓ | ✓ | +| v2.7.2 | v0.3.2 | ✓ | ✓ | +| v2.7.1 | v0.3.0 | ✓ | ✓ | +| v2.7.0 | v0.3.0 | ✓ | ✓ | ## 为什么我们需要它？ @@ -13,6 +40,58 @@ Rancher-Webhook 对于让 Rancher 保护集群免受恶意攻击并启用各种 Rancher 依赖 Rancher-Webhook 作为其功能的组成部分。如果没有 Webhook，Rancher 将不是一个完整的产品。 它为 Rancher 管理的集群提供了必要的保护，防止安全漏洞并确保集群的一致性和稳定性。 +## Webhook 验证哪些资源? + +你可以在 [webhook 仓库](https://github.com/rancher/webhook/blob/release/v0.4/docs.md)中找到 webhook 当前验证的资源列表。这些文档按组/版本（顶级标题）和资源（下一级标题）进行组织。可以通过查看与特定版本标签关联的 `docs.md` 文件来找到特定于一个版本的检查。请注意，`v0.3.6` 之前的 webhook 版本没有此文件。 + +## 绕过 Webhook + +有时，你必须绕过 Rancher 的 webhook 验证才能执行紧急还原操作或修复其他关键问题。避开操作是彻底的，这意味着在使用它时不会应用任何 webhook 验证或更改。不可能指定绕过某些验证，而让其他验证仍然可用。它们要么全部被绕过，要么全部处于活动状态。 + +:::danger + +Rancher 的 webhook 提供关键的安全保护。只有在所有其他选项都用尽之后，管理员才需要在特定情况下绕过 webhook。此外，应仔细控制绕过 webhook 的权限，切勿将该权限授予非管理员用户。 + +::: + +要绕过 webhook，请模拟 `rancher-webhook-sudo` 服务账号和 `system:masters` 组（两者都是必需的）： + +```bash +kubectl create -f example.yaml --as=system:serviceaccount:cattle-system:rancher-webhook-sudo --as-group=system:masters +``` + +## 自定义 Rancher-Webhook 配置 + +你可以通过 Helm 安装 Rancher-Webhook 时添加自定义 Helm values。在 Rancher-Webhook chart 的安装过程中，Rancher 会检查自定义的 Helm values。这些自定义值必须在 `cattle-system` 命名空间中，名称为 `rancher-config` 的 ConfigMap 的 data 属性下，增加 `rancher-webhook` 的配置定义。此键的值必须是有效的 YAML。 + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: rancher-config + namespace: cattle-system + labels: + app.kubernetes.io/part-of: "rancher" +data: + rancher-webhook: '{"port": 9553, "priorityClassName": "system-node-critical"}' +``` + +Rancher 会在检测到对 ConfigMap 值的更改时重新部署 Rancher-Webhook Chart。 + +### 在 Rancher 安装过程中自定义 Rancher-Webhook + +使用 Helm 安装 Rancher chart 时，可以在 local 集群中将自定义的 Helm values 添加到 Rancher-Webhook。Rancher-Webhook Chart 中的所有值都可以通过 `webhook` 名称下的嵌套变量访问。 + +这些值在安装过程中会同步到 `rancher-config` ConfigMap 中。 + +```bash +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + ... + --set webhook.port=9553 \ + --set webhook.priorityClassName="system-node-critical" +``` + ## 常见问题 ### 带有 Calico CNI 的 EKS 集群 @@ -24,8 +103,11 @@ Rancher 依赖 Rancher-Webhook 作为其功能的组成部分。如果没有 Web helm repo add rancher-charts https://charts.rancher.io helm upgrade --reuse-values rancher-webhook rancher-chart/rancher-webhook -n cattle-system --set global.hostNetwork=true ``` + **注意**：这个临时解决方法可能会违反环境的安全策略。此解决方法还要求主机网络上未使用端口 9443。 +**注意：** 默认情况下，Helm 使用 secrets。这是某些 webhook 版本验证用于存储信息的数据类型。在这种情况下，请使用 kubectl 更新 deployment 设置 `hostNetwork=true`，然后按照上述配置更新 webhook。 + ### 私有 GKE 集群 使用私有 GKE 集群时可能会发生错误，导致 Kubernetes API Server 无法与 Webhook 通信。以下错误消息可能会出现： @@ -33,4 +115,36 @@ helm upgrade --reuse-values rancher-webhook rancher-chart/rancher-webhook -n ca ``` Internal error occurred: failed calling webhook "rancher.cattle.io.namespaces.create-non-kubesystem": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s": context deadline exceeded ``` + 出现此问题的原因是防火墙规则限制了 API Server 与私有集群之间的通信。要解决此通信问题，你必须通过添加防火墙规则来允许 GKE Control Plane 通过端口 9443 与 Rancher-Webhook 进行通信。请参阅 [GKE 文档](https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules)，了解更新防火墙规则的详细信息和步骤。 + +### 由于 rancher-webhook 阻止访问导致应用部署失败 + +webhook 在 [namespaces](https://github.com/rancher/webhook/blob/release/v0.4/docs.md#psa-label-validation) 上提供额外的验证。其中一项验证可确保用户只有在具有适当权限的情况下才能更新 PSA 相关标签（`updatepsa` for `projects` in `management.cattle.io`）。这可能导致特定 operator（如 Tigera 或 Trident）在尝试部署带有 PSA 标签的命名空间时失败。有几种方法可以解决此问题： + +- 将应用程序配置为创建没有 PSA 标签的命名空间。如果用户希望将 PSA 应用于这些命名空间，则可以在配置后将它们添加到具有所需 PSA 的项目中。请参阅[设置 PSS 和 PSA 资源的文档](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/pod-security-standards.md)获取更具体的操作方法。 + - 这是首选选项，但并非所有应用程序都可以以这种方式进行配置。 +- 手动授予操作员管理命名空间下的 PSA 的权限。 + - 此选项将引入安全风险，因为运营商现在将能够为其有权访问的命名空间设置 PSA。这可能允许操作员部署特权 Pod，或通过其他方式实现集群接管。 +- 具有适当权限的用户帐户可以使用适当的配置预先创建命名空间。 + - 此选项取决于应用程序处理现有资源的能力。 + +## 特定版本的问题 + +**注意：** 以下是影响特定 Rancher/webhook 版本的高严重性问题的不完整列表。在大多数情况下，这些问题可以通过升级到更新的 Rancher 版本来解决。 + +### 回滚到不兼容的 Webhook 版本 + +**注意：** 这会影响回滚到 Rancher v2.7.5 或更早版本。 + +如果回滚到 Rancher v2.7.5 或更早版本，您可能会看到 webhook 版本太新，无法与运行 v2.7.5 之前版本的 Rancher 的下游集群兼容。这可能会导致各种不兼容问题。例如，项目成员可能无法创建命名空间。此外，当您回滚到下游集群中安装 webhook 之前的版本时，webhook 可能仍保持安装状态，这会导致类似的不兼容问题。 + +为了帮助缓解这些问题，您可以在回滚后运行 [adjust-downstream-webhook](https://github.com/rancherlabs/support-tools/tree/master/adjust-downstream-webhook) shell 脚本。该脚本为相应的 Rancher 版本选择并安装正确的 webhook 版本（或完全删除 webhook）。 + +### 项目用户无法创建命名空间 + +**注意：** 以下内容影响 Rancher v2.7.2 - v2.7.4。 + +项目用户可能无法在项目中创建命名空间，这包括项目所有者。此问题是由于 Rancher 自动将 webhook 升级到与当前安装的 Rancher 版本更新的版本不兼容而导致的。 + +为了帮助缓解这些问题，您可以在回滚后运行 [adjust-downstream-webhook](https://github.com/rancherlabs/support-tools/tree/master/adjust-downstream-webhook) shell 脚本。该脚本为相应的 Rancher 版本选择并安装正确的 webhook 版本（或完全删除 webhook）。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/single-node-rancher-in-docker/single-node-rancher-in-docker.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/single-node-rancher-in-docker/single-node-rancher-in-docker.md new file mode 100644 index 00000000000..e272daa1ca4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/single-node-rancher-in-docker/single-node-rancher-in-docker.md @@ -0,0 +1,9 @@ +--- +title: Docker 中的单节点 Rancher +--- + + + + + +以下文档将讨论 Docker 安装的 [HTTP 代理配置](http-proxy-configuration.md)和[高级选项](advanced-options.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/user-settings/user-settings.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/user-settings/user-settings.md new file mode 100644 index 00000000000..47ff9cb2518 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/reference-guides/user-settings/user-settings.md @@ -0,0 +1,19 @@ +--- +title: 用户设置 +--- + + + + + +在 Rancher 中，每个用户都有很多与登录相关的设置，例如个人偏好、API 密钥等。你可以从**用户设置**菜单中配置这些设置。你可以单击主菜单中的头像来打开此菜单。 + +![用户设置菜单](/img/user-settings.png) + +可用的用户设置包括： + +- [API & 密钥](api-keys.md)：如果你想以编程方式与 Rancher 交互，你需要一个 API 密钥。你可以按照本节中的说明获取密钥。 +- [云凭证](manage-cloud-credentials.md)：管理[节点模板](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md#节点模板)使用的云凭证，从而[为集群配置节点](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)。 +- [节点模板](manage-node-templates.md)：管理 [Rancher 用来为集群配置节点](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md)的模板。 +- [偏好设置](user-preferences.md)：设置 Rancher UI 的表面首选项。 +- 登出：结束你的用户会话。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/shared-files/_cluster-capabilities-table.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/shared-files/_cluster-capabilities-table.md index c418e1eaf54..c4b68aa5d8c 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/shared-files/_cluster-capabilities-table.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/shared-files/_cluster-capabilities-table.md @@ -2,13 +2,13 @@ | --- | --- | ---| ---|----| | [使用 kubectl 和 kubeconfig 文件来访问集群](../how-to-guides/new-user-guides/manage-clusters/access-clusters/use-kubectl-and-kubeconfig.md) | ✓ | ✓ | ✓ | ✓ | | [管理集群成员](../how-to-guides/new-user-guides/manage-clusters/access-clusters/add-users-to-clusters.md) | ✓ | ✓ | ✓ | ✓ | -| [编辑和升级集群](../pages-for-subheaders/cluster-configuration.md) | ✓ | ✓ | ✓ | ✓2 | +| [编辑和升级集群](../reference-guides/cluster-configuration/cluster-configuration.md) | ✓ | ✓ | ✓ | ✓2 | | [管理节点](../how-to-guides/new-user-guides/manage-clusters/nodes-and-node-pools.md) | ✓ | ✓ | ✓ | ✓3 | -| [管理持久卷和存储类](../pages-for-subheaders/create-kubernetes-persistent-storage.md) | ✓ | ✓ | ✓ | ✓ | +| [管理持久卷和存储类](../how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md) | ✓ | ✓ | ✓ | ✓ | | [管理项目、命名空间和工作负载](../how-to-guides/new-user-guides/manage-clusters/projects-and-namespaces.md) | ✓ | ✓ | ✓ | ✓ | -| [使用应用目录](../pages-for-subheaders/helm-charts-in-rancher.md) | ✓ | ✓ | ✓ | ✓ | -| 配置工具（[Alerts、Notifiers、Monitoring](../pages-for-subheaders/monitoring-and-alerting.md)、[Logging](../pages-for-subheaders/logging.md) 和 [Istio](../pages-for-subheaders/istio.md)） | ✓ | ✓ | ✓ | ✓ | -| [运行安全扫描](../pages-for-subheaders/cis-scan-guides.md) | ✓ | ✓ | ✓ | ✓ | +| [使用应用目录](../how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md) | ✓ | ✓ | ✓ | ✓ | +| 配置工具（[Alerts、Notifiers、Monitoring](../integrations-in-rancher/monitoring-and-alerting/monitoring-and-alerting.md)、[Logging](../integrations-in-rancher/logging/logging.md) 和 [Istio](../integrations-in-rancher/istio/istio.md)） | ✓ | ✓ | ✓ | ✓ | +| [运行安全扫描](../how-to-guides/advanced-user-guides/cis-scan-guides/cis-scan-guides.md) | ✓ | ✓ | ✓ | ✓ | | [轮换证书](../how-to-guides/new-user-guides/manage-clusters/rotate-certificates.md) | ✓ | ✓ | | | | [备份](../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/back-up-rancher-launched-kubernetes-clusters.md)和[恢复](../how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher-launched-kubernetes-clusters-from-backup.md) Rancher 启动的集群 | ✓ | ✓ | | ✓4 | | [在 Rancher 无法访问集群时清理 Kubernetes 组件](../how-to-guides/new-user-guides/manage-clusters/clean-cluster-nodes.md) | ✓ | | | | diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/troubleshooting/kubernetes-components/kubernetes-components.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/troubleshooting/kubernetes-components/kubernetes-components.md new file mode 100644 index 00000000000..38bcba2810d --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/troubleshooting/kubernetes-components/kubernetes-components.md @@ -0,0 +1,21 @@ +--- +title: Kubernetes 组件 +--- + + + + + +本文列出的命令和步骤适用于 [Rancher 启动的 Kubernetes](../../how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md) 集群上的核心 Kubernetes 组件。 + +本文包括以下类别的故障排除提示： + +- [etcd 节点故障排除](troubleshooting-etcd-nodes.md) +- [Controlplane 节点故障排除](troubleshooting-controlplane-nodes.md) +- [nginx-proxy 节点故障排除](troubleshooting-nginx-proxy.md) +- [Worker 节点和通用组件故障排除](troubleshooting-worker-nodes-and-generic-components.md) + +## Kubernetes 组件图 + +![集群图](/img/clusterdiagram.svg)
+线条表示组件之间的通信。而颜色纯粹用于视觉辅助。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/troubleshooting/kubernetes-components/troubleshooting-etcd-nodes.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/troubleshooting/kubernetes-components/troubleshooting-etcd-nodes.md index 72d6b470bfc..d64215cebd2 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.7/troubleshooting/kubernetes-components/troubleshooting-etcd-nodes.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.7/troubleshooting/kubernetes-components/troubleshooting-etcd-nodes.md @@ -2,6 +2,10 @@ title: etcd 节点故障排除 --- + + + + 本文介绍了对具有 `etcd` 角色的节点进行故障排除的命令和提示。 @@ -15,8 +19,8 @@ docker ps -a -f=name=etcd$ 输出示例： ``` -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -605a124503b9 rancher/coreos-etcd:v3.2.18 "/usr/local/bin/et..." 2 hours ago Up 2 hours etcd +CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES +d26adbd23643 rancher/mirrored-coreos-etcd:v3.5.7 "/usr/local/bin/etcd…" 30 minutes ago Up 30 minutes etcd ``` ## etcd 容器日志记录 @@ -47,30 +51,13 @@ docker logs etcd docker exec etcd etcdctl member list ``` -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list" -``` - -输出示例： -``` -xxx, started, etcd-xxx, https://IP:2380, https://IP:2379,https://IP:4001 -xxx, started, etcd-xxx, https://IP:2380, https://IP:2379,https://IP:4001 -xxx, started, etcd-xxx, https://IP:2380, https://IP:2379,https://IP:4001 -``` - ### 检查端点状态 `RAFT TERM` 的值应该是相等的，而且 `RAFT INDEX` 相差不能太大。 命令： ``` -docker exec -e ETCDCTL_ENDPOINTS=$(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") etcd etcdctl endpoint status --write-out table -``` - -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -docker exec etcd etcdctl endpoint status --endpoints=$(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") --write-out table +docker exec -e ETCDCTL_ENDPOINTS=$(docker exec etcd etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') etcd etcdctl endpoint status --write-out table ``` 输出示例： @@ -78,9 +65,9 @@ docker exec etcd etcdctl endpoint status --endpoints=$(docker exec etcd /bin/sh +-----------------+------------------+---------+---------+-----------+-----------+------------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | +-----------------+------------------+---------+---------+-----------+-----------+------------+ -| https://IP:2379 | 333ef673fc4add56 | 3.2.18 | 24 MB | false | 72 | 66887 | -| https://IP:2379 | 5feed52d940ce4cf | 3.2.18 | 24 MB | true | 72 | 66887 | -| https://IP:2379 | db6b3bdb559a848d | 3.2.18 | 25 MB | false | 72 | 66887 | +| https://IP:2379 | 333ef673fc4add56 | 3.5.7 | 24 MB | false | 72 | 66887 | +| https://IP:2379 | 5feed52d940ce4cf | 3.5.7 | 24 MB | true | 72 | 66887 | +| https://IP:2379 | db6b3bdb559a848d | 3.5.7 | 25 MB | false | 72 | 66887 | +-----------------+------------------+---------+---------+-----------+-----------+------------+ ``` @@ -88,12 +75,7 @@ docker exec etcd etcdctl endpoint status --endpoints=$(docker exec etcd /bin/sh 命令： ``` -docker exec -e ETCDCTL_ENDPOINTS=$(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") etcd etcdctl endpoint health -``` - -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -docker exec etcd etcdctl endpoint health --endpoints=$(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") +docker exec -e ETCDCTL_ENDPOINTS=$(docker exec etcd etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') etcd etcdctl endpoint health ``` 输出示例： @@ -107,17 +89,9 @@ https://IP:2379 is healthy: successfully committed proposal: took = 2.451201ms 命令： ``` -for endpoint in $(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5"); do +for endpoint in $(docker exec etcd etcdctl member list | cut -d, -f5); do echo "Validating connection to ${endpoint}/health" - docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/health" -done -``` - -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -for endpoint in $(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f5"); do - echo "Validating connection to ${endpoint}/health"; - docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/health" + docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -w "\n" --cacert $(docker inspect -f '{{range $index, $value := .Config.Env}}{{if eq (index (split $value "=") 0) "ETCDCTL_CACERT" }}{{range $i, $part := (split $value "=")}}{{if gt $i 1}}{{print "="}}{{end}}{{if gt $i 0}}{{print $part}}{{end}}{{end}}{{end}}{{end}}' etcd) --cert $(docker inspect -f '{{range $index, $value := .Config.Env}}{{if eq (index (split $value "=") 0) "ETCDCTL_CERT" }}{{range $i, $part := (split $value "=")}}{{if gt $i 1}}{{print "="}}{{end}}{{if gt $i 0}}{{print $part}}{{end}}{{end}}{{end}}{{end}}' etcd) --key $(docker inspect -f '{{range $index, $value := .Config.Env}}{{if eq (index (split $value "=") 0) "ETCDCTL_KEY" }}{{range $i, $part := (split $value "=")}}{{if gt $i 1}}{{print "="}}{{end}}{{if gt $i 0}}{{print $part}}{{end}}{{end}}{{end}}{{end}}' etcd) "${endpoint}/health" done ``` @@ -135,28 +109,20 @@ Validating connection to https://IP:2379/health 命令： ``` -for endpoint in $(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f4"); do +for endpoint in $(docker exec etcd etcdctl member list | cut -d, -f4); do echo "Validating connection to ${endpoint}/version"; - docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl --http1.1 -s -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/version" -done -``` - -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -for endpoint in $(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f4"); do - echo "Validating connection to ${endpoint}/version"; - docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl --http1.1 -s -w "\n" --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) "${endpoint}/version" + docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl --http1.1 -s -w "\n" --cacert $(docker inspect -f '{{range $index, $value := .Config.Env}}{{if eq (index (split $value "=") 0) "ETCDCTL_CACERT" }}{{range $i, $part := (split $value "=")}}{{if gt $i 1}}{{print "="}}{{end}}{{if gt $i 0}}{{print $part}}{{end}}{{end}}{{end}}{{end}}' etcd) --cert $(docker inspect -f '{{range $index, $value := .Config.Env}}{{if eq (index (split $value "=") 0) "ETCDCTL_CERT" }}{{range $i, $part := (split $value "=")}}{{if gt $i 1}}{{print "="}}{{end}}{{if gt $i 0}}{{print $part}}{{end}}{{end}}{{end}}{{end}}' etcd) --key $(docker inspect -f '{{range $index, $value := .Config.Env}}{{if eq (index (split $value "=") 0) "ETCDCTL_KEY" }}{{range $i, $part := (split $value "=")}}{{if gt $i 1}}{{print "="}}{{end}}{{if gt $i 0}}{{print $part}}{{end}}{{end}}{{end}}{{end}}' etcd) "${endpoint}/version" done ``` 输出示例： ``` Validating connection to https://IP:2380/version -{"etcdserver":"3.2.18","etcdcluster":"3.2.0"} +{"etcdserver":"3.5.7","etcdcluster":"3.5.0"} Validating connection to https://IP:2380/version -{"etcdserver":"3.2.18","etcdcluster":"3.2.0"} +{"etcdserver":"3.5.7","etcdcluster":"3.5.0"} Validating connection to https://IP:2380/version -{"etcdserver":"3.2.18","etcdcluster":"3.2.0"} +{"etcdserver":"3.5.7","etcdcluster":"3.5.0"} ``` ## etcd 告警 @@ -168,11 +134,6 @@ etcd 会触发告警（例如空间不足时）。 docker exec etcd etcdctl alarm list ``` -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT alarm list" -``` - 触发 NOSPACE 告警的输出示例： ``` memberID:x alarm:NOSPACE @@ -199,12 +160,6 @@ rev=$(docker exec etcd etcdctl endpoint status --write-out json | egrep -o '"rev docker exec etcd etcdctl compact "$rev" ``` -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -rev=$(docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT endpoint status --write-out json | egrep -o '\"revision\":[0-9]*' | egrep -o '[0-9]*'") -docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT compact \"$rev\"" -``` - 输出示例： ``` compacted revision xxx @@ -214,12 +169,7 @@ compacted revision xxx 命令： ``` -docker exec -e ETCDCTL_ENDPOINTS=$(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") etcd etcdctl defrag -``` - -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -docker exec etcd sh -c "etcdctl defrag --endpoints=$(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','")" +docker exec -e ETCDCTL_ENDPOINTS=$(docker exec etcd etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') etcd etcdctl defrag ``` 输出示例： @@ -233,12 +183,7 @@ Finished defragmenting etcd member[https://IP:2379] 命令： ``` -docker exec -e ETCDCTL_ENDPOINTS=$(docker exec etcd /bin/sh -c "etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") etcd etcdctl endpoint status --write-out table -``` - -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -docker exec etcd sh -c "etcdctl endpoint status --endpoints=$(docker exec etcd /bin/sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ','") --write-out table" +docker exec -e ETCDCTL_ENDPOINTS=$(docker exec etcd etcdctl member list | cut -d, -f5 | sed -e 's/ //g' | paste -sd ',') etcd etcdctl endpoint status --write-out table ``` 输出示例： @@ -246,9 +191,9 @@ docker exec etcd sh -c "etcdctl endpoint status --endpoints=$(docker exec etcd / +-----------------+------------------+---------+---------+-----------+-----------+------------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | +-----------------+------------------+---------+---------+-----------+-----------+------------+ -| https://IP:2379 | e973e4419737125 | 3.2.18 | 553 kB | false | 32 | 2449410 | -| https://IP:2379 | 4a509c997b26c206 | 3.2.18 | 553 kB | false | 32 | 2449410 | -| https://IP:2379 | b217e736575e9dd3 | 3.2.18 | 553 kB | true | 32 | 2449410 | +| https://IP:2379 | e973e4419737125 | 3.5.7 | 553 kB | false | 32 | 2449410 | +| https://IP:2379 | 4a509c997b26c206 | 3.5.7 | 553 kB | false | 32 | 2449410 | +| https://IP:2379 | b217e736575e9dd3 | 3.5.7 | 553 kB | true | 32 | 2449410 | +-----------------+------------------+---------+---------+-----------+-----------+------------+ ``` @@ -263,13 +208,6 @@ docker exec etcd etcdctl alarm disarm docker exec etcd etcdctl alarm list ``` -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT alarm list" -docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT alarm disarm" -docker exec etcd sh -c "etcdctl --endpoints=\$ETCDCTL_ENDPOINT alarm list" -``` - 输出示例： ``` docker exec etcd etcdctl alarm list @@ -307,11 +245,6 @@ services: docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -XPUT -d '{"Level":"DEBUG"}' --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) $(docker exec etcd printenv ETCDCTL_ENDPOINTS)/config/local/log ``` -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -XPUT -d '{"Level":"DEBUG"}' --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) $(docker exec etcd printenv ETCDCTL_ENDPOINT)/config/local/log -``` - 要将日志级别重置回默认值 (`INFO`)，你可以使用以下命令。 命令： @@ -319,11 +252,6 @@ docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{ docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -XPUT -d '{"Level":"INFO"}' --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) $(docker exec etcd printenv ETCDCTL_ENDPOINTS)/config/local/log ``` -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{{ if eq .Destination "/etc/kubernetes" }}{{ .Source }}{{ end }}{{ end }}')/ssl:/etc/kubernetes/ssl:ro appropriate/curl -s -XPUT -d '{"Level":"INFO"}' --cacert $(docker exec etcd printenv ETCDCTL_CACERT) --cert $(docker exec etcd printenv ETCDCTL_CERT) --key $(docker exec etcd printenv ETCDCTL_KEY) $(docker exec etcd printenv ETCDCTL_ENDPOINT)/config/local/log -``` - ## etcd 内容 如果要查看 etcd 的内容，你可以查看流事件，也可以直接查询 etcd。详情请参阅以下示例。 @@ -335,11 +263,6 @@ docker run --net=host -v $(docker inspect kubelet --format '{{ range .Mounts }}{ docker exec etcd etcdctl watch --prefix /registry ``` -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -docker exec etcd etcdctl --endpoints=\$ETCDCTL_ENDPOINT watch --prefix /registry -``` - 如果你只想查看受影响的键（而不是二进制数据），你可以将 `| grep -a ^/registry` 尾附到该命令来过滤键。 ### 直接查询 etcd @@ -349,11 +272,6 @@ docker exec etcd etcdctl --endpoints=\$ETCDCTL_ENDPOINT watch --prefix /registry docker exec etcd etcdctl get /registry --prefix=true --keys-only ``` -如果 etcd 版本低于 3.3.x（Kubernetes 1.13.x 及更低版本）且添加节点时指定了 `--internal-address`，则使用以下命令： -``` -docker exec etcd etcdctl --endpoints=\$ETCDCTL_ENDPOINT get /registry --prefix=true --keys-only -``` - 你可以使用以下命令来处理数据，从而获取每个键的计数摘要： ``` diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/faq/general-faq.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/faq/general-faq.md index 45d4ae835ae..5cf116534af 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/faq/general-faq.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/faq/general-faq.md @@ -2,68 +2,50 @@ title: 一般常见问题解答 --- + + + + 本文包含了用户常见的 Rancher 2.x 问题。 有关常见技术问题，请参阅[常见技术问题解答](technical-items.md)。 -
- -**Rancher 2.x 支持 Docker Swarm 和 Mesos 作为环境类型吗？** +## Rancher 2.x 支持 Docker Swarm 和 Mesos 作为环境类型吗？ 如果你在 Rancher 2.x 中创建环境，Swarm 和 Mesos 将不再是可选的标准选项。但是，Swarm 和 Mesos 还能继续作为可以部署的商店应用程序。这是一个艰难的决定，但这是大势所趋。比如说，15,000 多个集群可能只有大约 200 个在运行 Swarm。 -
+## 是否可以使用 Rancher 2.x 管理 Azure Kubernetes 服务？ -**是否可以使用 Rancher 2.x 管理 Azure Kubernetes 服务？** +是的。请参阅我们的[集群管理]((../how-to-guides/new-user-guides/manage-clusters/manage-clusters.md))指南，了解 AKS 上可用的 Rancher 功能，以及相关的 [AKS 的文档](../getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rancher-on-aks.md) -是的。 +## Rancher 是否支持 Windows？ -
+Rancher 支持 Windows Server 1809 容器。有关如何使用 Windows Worker 节点设置集群的详细信息，请参阅[为 Windows 配置自定义集群](../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md)。 -**Rancher 是否支持 Windows？** - -Rancher 支持 Windows Server 1809 容器。有关如何使用 Windows Worker 节点设置集群的详细信息，请参阅[为 Windows 配置自定义集群](../pages-for-subheaders/use-windows-clusters.md)。 - -
- -**Rancher 是否支持 Istio？** +## Rancher 是否支持 Istio？ Rancher 支持 [Istio](../pages-for-subheaders/istio.md)。 -此外，Istio 是在我们的微型 PaaS “Rio” 中实现的，它可以运行在 Rancher 2.x 以及任何符合 CNCF 的 Kubernetes 集群上。详情请参阅[这里](https://rio.io/)。 - -
- -**Rancher 2.x 是否支持使用 Hashicorp 的 Vault 来存储密文？** +## Rancher 2.x 是否支持使用 Hashicorp 的 Vault 来存储密文？ 密文管理已在我们的 roadmap 上，但我们尚未将该功能分配给特定版本。 -
- -**Rancher 2.x 是否也支持 RKT 容器？** +## Rancher 2.x 是否也支持 RKT 容器？ 目前，我们只支持 Docker。 -
- -**Rancher 2.x 是否支持将 Calico、Contiv、Contrail、Flannel、Weave net 等网络插件用于嵌入和已注册的 Kubernetes？** +## Rancher 2.x 是否支持将 Calico、Contiv、Contrail、Flannel、Weave net 等网络插件用于嵌入和已注册的 Kubernetes？ Rancher 开箱即用地为 Kubernetes 集群提供了几个 CNI 网络插件，分别是 Canal、Flannel、Calico 和 Weave。有关官方支持的详细信息，请参阅 [Rancher 支持矩阵](https://rancher.com/support-maintenance-terms/)。 -
- -**Rancher 是否计划支持 Traefik？** +## Rancher 是否计划支持 Traefik？ 目前，我们不打算提供嵌入式 Traefik 支持，但我们仍在探索负载均衡方案。 -
- -**我可以将 OpenShift Kubernetes 集群导入 2.x 吗？** +## 我可以将 OpenShift Kubernetes 集群导入 2.x 吗？ 我们的目标是运行任何上游 Kubernetes 集群。因此，Rancher 2.x 应该可以与 OpenShift 一起使用，但我们尚未对此进行测试。 -
- -**Rancher 会集成 Longhorn 吗？** +## Rancher 会集成 Longhorn 吗？ 是的。Longhorn 已集成到 Rancher 2.5+ 中。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md index c924cc3c797..ca0cdcc3a2f 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/getting-started/installation-and-upgrade/installation-references/helm-chart-options.md @@ -159,7 +159,7 @@ Rancher 的一些功能（Helm Chart）需要使用互联网才能使用。你 不包括敏感数据时，可以使用 `proxy` 或 `extraEnv` Chart 选项。使用 `extraEnv` 时将忽略 `noProxy` Helm 选项。因此，`NO_PROXY` 环境变量也必须设置为 `extraEnv`。 -以下是使用 `extraEnv` Chart 选项设置代理的示例： +以下是使用 `proxy` Chart 选项设置代理的示例： ```plain --set proxy="http:///" diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md index 11760970b23..eeb6cb521f0 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/restore-rancher.md @@ -4,38 +4,23 @@ title: 还原 Rancher 本页概述了如何使用 Rancher 执行恢复。 -:::note 重要提示： +在以下情况下，请按照本页中的说明进行操作： +- 正在运行的 Rancher 实例与备份时的版本相同。 +- 上游（本地）集群与备份的位置相同。 -* 请按照此页面上的说明在已备份的同一集群上还原 Rancher。要把 Rancher 迁移到新集群，请参照步骤[迁移 Rancher](migrate-rancher-to-new-cluster.md)。 -* 在使用相同设置还原 Rancher 时，operator 将在还原开始时缩减 Rancher deployment，还原完成后又会扩展 deployment。因此，Rancher 在还原期间将不可用。 +:::note 重要提示 + +在使用相同设置还原 Rancher 时，operator 将在还原开始时缩减 Rancher deployment，还原完成后又会扩展 deployment。因此，Rancher 在还原期间将不可用。 + +::: + +:::tip + +* 按照以下步骤[迁移 Rancher](migrate-rancher-to-new-cluster.md)。 * 如果你需要在升级后将 Rancher 还原到先前版本，请参见[回滚](../../../getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/rollbacks.md)。 ::: -## 使用 Rancher 2.6.4+ 进行回滚的其他步骤 - -Rancher v2.6.4 将 cluster-api 模块从 v0.4.4 升级到 v1.0.2。反过来，cluster-api 的 v1.0.2 版本将集群 API 的自定义资源定义 (CRD) 从 `cluster.x-k8s.io/v1alpha4` 升级到 `cluster.x-k8s.io/v1beta1`。当你尝试将 Rancher v2.6.4 回滚到以前版本的 Rancher v2.6.x 时，CRD 升级到 v1beta1 会导致回滚失败。这是因为使用旧 apiVersion (v1alpha4) 的 CRD 与 v1beta1 不兼容。 - -要避免回滚失败，你需要在尝试恢复操作或回滚**之前**运行以下 Rancher 脚本： - -* `verify.sh`：检查集群中是否有任何与 Rancher 相关的资源。 -* `cleanup.sh`：清理集群。 - -有关详细信息和源代码，请参阅 [rancher/rancher-cleanup repo](https://github.com/rancher/rancher-cleanup)。 - -:::caution - -`cleanup.sh` 运行的时候会有停机时间，这是因为脚本会删除 Rancher 创建的资源。 - -::: - -### 从 v2.6.4+ 回滚到较低版本的 v2.6.x - -1. 按照[说明](https://github.com/rancher/rancher-cleanup/blob/main/README.md)运行脚本。 -1. 按照[说明](../backup-restore-and-disaster-recovery/migrate-rancher-to-new-cluster.md)在现有集群上安装 rancher-backup Helm Chart 并恢复之前的状态。 - 1. 省略步骤 3。 - 1. 执行到步骤 4 时，在要回滚到的 local 集群上安装 Rancher 2.6.x 版本。 - ### 创建 Restore 自定义资源 还原是通过创建 Restore 自定义资源实现的。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md index aa10ce27651..1bd4160152e 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/cluster-api.md @@ -9,6 +9,6 @@ title: Cluster API (CAPI) 与 Rancher Turtles [Rancher Turtles](https://turtles.docs.rancher.com/) 是一个 [Rancher 扩展](../rancher-extensions.md)，通过提供 Cluster API (CAPI) 和 Rancher 之间的集成来管理配置的 Kubernetes 集群的生命周期。使用 Rancher Turtles，你可以： - 通过在 CAPI 配置的集群中安装 Rancher Cluster Agent，将 CAPI 集群导入 Rancher。 -- 配置 [CAPI Operator](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values)。 +- 配置 [CAPI Operator](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values#cluster-api-operator-values)。 [概述](./overview.md)部分介绍了安装选项、Rancher Turtles 架构和简要 Demo。有关详细信息，请参阅 [Rancher Turtles 文档](https://turtles.docs.rancher.com/)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/overview.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/overview.md index 2981c2199ef..5da4470bfd6 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/overview.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/integrations-in-rancher/cluster-api/overview.md @@ -12,6 +12,12 @@ title: 概述 ![overview](/img/30000ft_view.png) +## 安全 + +[SLSA](https://slsa.dev/spec/v1.0/about) 是一套由行业共识制定的可逐步采用的供应链安全指南。SLSA 制定的规范对软件生产者和消费者都很有用：生产者可以遵循 SLSA 的指导方针，使他们的软件供应链更加安全，消费者可以使用 SLSA 来决定是否信任软件包。 + +Rancher Turtles 满足 [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) 对适当的构建平台、一致的构建过程和来源分布的要求。更多信息请参阅 [Rancher Turtles 安全](https://turtles.docs.rancher.com/security/slsa)文档。 + ## 先决条件 在 Rancher 环境中安装 Rancher Turtles 之前，你必须禁用 Rancher 的 `embedded-cluster-api` 功能。这还包括清理 Rancher 专用的 webhook，否则这些 webhook 将与 CAPI 的 webhook 冲突。 @@ -21,6 +27,45 @@ title: 概述 - 禁用 Rancher 中的 `embedded-cluster-api` 功能。 - 删除不再需要的 `mutating-webhook-configuration` 和 `validating-webhook-configuration` webhook。 +这些 webhook 也可以通过 Rancher UI 删除： + +1. 点击左上角 **☰ > 集群管理**。 +1. 选择你的 local 集群。 +1. 在左侧导航菜单，选择 **More Resources** > **Admission**。 +1. 在下拉菜单中，选择资源页面的 `MutatingWebhookConfiguration` 和 `ValidatingWebhookConfiguration`。 +1. 在相应的资源页面上，点击 `mutating-webhook-configuration` and `validating-webhook-configuration` 后面的 **⋮** 然后选择 **删除**。 + +还可以通过在 **Resource Search** 字段中输入 webhook 的名称来访问到具体的 webhook。 + +以下的 `kubectl` 命令可以手动删除必要的 webhook： + +```console +kubectl delete mutatingwebhookconfiguration.admissionregistration.k8s.io mutating-webhook-configuration +``` + +```console +kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io validating-webhook-configuration +``` + +使用以下示例从控制台禁用 `embedded-cluster-api` 功能： + +1. 创建一个 `feature.yaml` 文件，将 `embedded-cluster-api` 设置为 false： + +```yaml title="feature.yaml" +apiVersion: management.cattle.io/v3 +kind: Feature +metadata: + name: embedded-cluster-api +spec: + value: false +``` + +2. 使用 `kubectl` 将 `feature.yaml` 文件应用到集群： + +```bash +kubectl apply -f feature.yaml +``` + ## 安装 Rancher Turtles Operator 你可以通过 Rancher UI 或使用 Helm 安装 Rancher Turtles operator。对于大多数环境推荐使用第一种方法。 @@ -47,7 +92,7 @@ title: 概述 1. 点击 **Rancher Turtles - the Cluster API Extension**。 1. 点击 **Install > Next > Install**. -此过程使用 Helm chart 的默认值，这些值适用于大部分的安装场景。如果你的配置需要覆盖其中一些默认值，你可以在安装期间通过 Rancher UI 指定这些值，也可以通过 [Helm 手动安装 Chart](#通过-helm-安装)。有关可用的 values 设置的详细信息，请参阅 Rancher Turtles 的 [Helm chart 参考指南](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values)。 +此过程使用 Helm chart 的默认值，这些值适用于大部分的安装场景。如果你的配置需要覆盖其中一些默认值，你可以在安装期间通过 Rancher UI 指定这些值，也可以通过 [Helm 手动安装 Chart](#通过-helm-安装)。有关可用的 values 设置的详细信息，请参阅 Rancher Turtles 的 [Helm chart 参考指南](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values)。 安装可能需要几分钟时间，安装完成后，你可以在集群中看到以下新部署： @@ -132,7 +177,7 @@ stringData: :::info -有关 chart 支持的 values 及其用法的详细信息，请参阅 [Helm chart 选项](https://turtles.docs.rancher.com/docs/reference-guides/rancher-turtles-chart/values) +有关 chart 支持的 values 及其用法的详细信息，请参阅 [Helm chart 选项](https://turtles.docs.rancher.com/reference-guides/rancher-turtles-chart/values) ::: @@ -140,7 +185,7 @@ stringData: :::note -请记住，如果使用此安装选项，你必须自行管理 CAPI Operator 的安装。你可以参照 Rancher Turtles 文档中的 [CAPI Operator 指南](https://turtles.docs.rancher.com/docs/tasks/capi-operator/intro) +请记住，如果使用此安装选项，你必须自行管理 CAPI Operator 的安装。你可以参照 Rancher Turtles 文档中的 [CAPI Operator 指南](https://turtles.docs.rancher.com/tasks/capi-operator/intro) ::: @@ -213,9 +258,3 @@ spec: ```bash kubectl apply -f feature.yaml ``` - -## 安全 - -[SLSA](https://slsa.dev/spec/v1.0/about) 是一套由行业共识制定的可逐步采用的供应链安全指南。SLSA 制定的规范对软件生产者和消费者都很有用：生产者可以遵循 SLSA 的指导方针，使他们的软件供应链更加安全，消费者可以使用 SLSA 来决定是否信任软件包。 - -Rancher Turtles 满足 [SLSA Level 3](https://slsa.dev/spec/v1.0/levels#build-l3) 对适当的构建平台、一致的构建过程和来源分布的要求。更多信息请参阅 [Rancher Turtles 安全](https://turtles.docs.rancher.com/docs/security/slsa)文档。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-webhook.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-webhook.md index caa0335ca2a..e8a77888233 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-webhook.md +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.8/reference-guides/rancher-webhook.md @@ -2,10 +2,28 @@ title: Rancher Webhook --- + + + + Rancher-Webhook 是 Rancher 的重要组件，它与 Kubernetes 结合使用，用于增强安全性并为 Rancher 管理的集群启用关键功能。 如 [Kubernetes 文档](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/)中所述，它与 Kubernetes 的可扩展准入控制器集成，允许 Rancher-Webhook 检查发送到 Kubernetes API Server 的特定请求，添加自定义和 Rancher 相关的验证，以及 Rancher 相关请求的变化。Rancher-Webhook 使用 `rancher.cattle.io` `ValidatingWebhookConfiguration` 和 `rancher.cattle.io` `MutatingWebhookConfiguration` 管理要验证的资源，并覆盖任何手动编辑。 -Rancher 将 Rancher-Webhook 作为单独的 deployment 和服务部署在 local 和下游集群中。Rancher 使用 Helm 管理 Rancher-Webhook。需要注意的是，Rancher 可能会覆盖用户对 Helm 版本所做的修改。 + +Rancher 将 Rancher-Webhook 作为单独的 deployment 和服务部署在 local 和下游集群中。Rancher 使用 Helm 管理 Rancher-Webhook。需要注意的是，Rancher 可能会覆盖用户对 Helm 版本所做的修改。要安全地修改这些值，请参阅[自定义 Rancher-Webhook 配置](#自定义-rancher-webhook-配置) + +每个 Rancher 版本都设计为与某个具体版本的 Webhook 兼容，为方便起见，下面提供了各版本的兼容列表。 + +**注意：** Rancher 负责管理 webhook 的部署和升级。在多数情况下，不需要用户干预来确保 webhook 版本与你正在运行的 Rancher 版本兼容。 + + + +| Rancher Version | Webhook Version | Availability in Prime | Availability in Community | +| --------------- | --------------- | --------------------- | ------------------------- | +| v2.8.3 | v0.4.3 | ✓ | ✓ | +| v2.8.2 | v0.4.2 | ✓ | ✓ | +| v2.8.1 | v0.4.2 | ✓ | ✓ | +| v2.8.0 | v0.4.2 | ✗ | ✓ | ## 为什么我们需要它？ @@ -13,6 +31,58 @@ Rancher-Webhook 对于让 Rancher 保护集群免受恶意攻击并启用各种 Rancher 依赖 Rancher-Webhook 作为其功能的组成部分。如果没有 Webhook，Rancher 将不是一个完整的产品。 它为 Rancher 管理的集群提供了必要的保护，防止安全漏洞并确保集群的一致性和稳定性。 +## Webhook 验证哪些资源? + +你可以在 [webhook 仓库](https://github.com/rancher/webhook/blob/release/v0.4/docs.md)中找到 webhook 当前验证的资源列表。这些文档按组/版本（顶级标题）和资源（下一级标题）进行组织。可以通过查看与特定版本标签关联的 `docs.md` 文件来找到特定于一个版本的检查。请注意，`v0.3.6` 之前的 webhook 版本没有此文件。 + +## 绕过 Webhook + +有时，你必须绕过 Rancher 的 webhook 验证才能执行紧急还原操作或修复其他关键问题。避开操作是彻底的，这意味着在使用它时不会应用任何 webhook 验证或更改。不可能指定绕过某些验证，而让其他验证仍然可用。它们要么全部被绕过，要么全部处于活动状态。 + +:::danger + +Rancher 的 webhook 提供关键的安全保护。只有在所有其他选项都用尽之后，管理员才需要在特定情况下绕过 webhook。此外，应仔细控制绕过 webhook 的权限，切勿将该权限授予非管理员用户。 + +::: + +要绕过 webhook，请模拟 `rancher-webhook-sudo` 服务账号和 `system:masters` 组（两者都是必需的）： + +```bash +kubectl create -f example.yaml --as=system:serviceaccount:cattle-system:rancher-webhook-sudo --as-group=system:masters +``` + +## 自定义 Rancher-Webhook 配置 + +你可以通过 Helm 安装 Rancher-Webhook 时添加自定义 Helm values。在 Rancher-Webhook chart 的安装过程中，Rancher 会检查自定义的 Helm values。这些自定义值必须在 `cattle-system` 命名空间中，名称为 `rancher-config` 的 ConfigMap 的 data 属性下，增加 `rancher-webhook` 的配置定义。此键的值必须是有效的 YAML。 + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: rancher-config + namespace: cattle-system + labels: + app.kubernetes.io/part-of: "rancher" +data: + rancher-webhook: '{"port": 9553, "priorityClassName": "system-node-critical"}' +``` + +Rancher 会在检测到对 ConfigMap 值的更改时重新部署 Rancher-Webhook Chart。 + +### 在 Rancher 安装过程中自定义 Rancher-Webhook + +使用 Helm 安装 Rancher chart 时，可以在 local 集群中将自定义的 Helm values 添加到 Rancher-Webhook。Rancher-Webhook Chart 中的所有值都可以通过 `webhook` 名称下的嵌套变量访问。 + +这些值在安装过程中会同步到 `rancher-config` ConfigMap 中。 + +```bash +helm install rancher rancher-/rancher \ + --namespace cattle-system \ + ... + --set webhook.port=9553 \ + --set webhook.priorityClassName="system-node-critical" +``` + ## 常见问题 ### 带有 Calico CNI 的 EKS 集群 @@ -24,8 +94,11 @@ Rancher 依赖 Rancher-Webhook 作为其功能的组成部分。如果没有 Web helm repo add rancher-charts https://charts.rancher.io helm upgrade --reuse-values rancher-webhook rancher-chart/rancher-webhook -n cattle-system --set global.hostNetwork=true ``` + **注意**：这个临时解决方法可能会违反环境的安全策略。此解决方法还要求主机网络上未使用端口 9443。 +**注意：** 默认情况下，Helm 使用 secrets。这是某些 webhook 版本验证用于存储信息的数据类型。在这种情况下，请使用 kubectl 更新 deployment 设置 `hostNetwork=true`，然后按照上述配置更新 webhook。 + ### 私有 GKE 集群 使用私有 GKE 集群时可能会发生错误，导致 Kubernetes API Server 无法与 Webhook 通信。以下错误消息可能会出现： @@ -33,4 +106,36 @@ helm upgrade --reuse-values rancher-webhook rancher-chart/rancher-webhook -n ca ``` Internal error occurred: failed calling webhook "rancher.cattle.io.namespaces.create-non-kubesystem": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/namespaces?timeout=10s": context deadline exceeded ``` + 出现此问题的原因是防火墙规则限制了 API Server 与私有集群之间的通信。要解决此通信问题，你必须通过添加防火墙规则来允许 GKE Control Plane 通过端口 9443 与 Rancher-Webhook 进行通信。请参阅 [GKE 文档](https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules)，了解更新防火墙规则的详细信息和步骤。 + +### 由于 rancher-webhook 阻止访问导致应用部署失败 + +webhook 在 [namespaces](https://github.com/rancher/webhook/blob/release/v0.4/docs.md#psa-label-validation) 上提供额外的验证。其中一项验证可确保用户只有在具有适当权限的情况下才能更新 PSA 相关标签（`updatepsa` for `projects` in `management.cattle.io`）。这可能导致特定 operator（如 Tigera 或 Trident）在尝试部署带有 PSA 标签的命名空间时失败。有几种方法可以解决此问题： + +- 将应用程序配置为创建没有 PSA 标签的命名空间。如果用户希望将 PSA 应用于这些命名空间，则可以在配置后将它们添加到具有所需 PSA 的项目中。请参阅[设置 PSS 和 PSA 资源的文档](../how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/pod-security-standards.md)获取更具体的操作方法。 + - 这是首选选项，但并非所有应用程序都可以以这种方式进行配置。 +- 手动授予操作员管理命名空间下的 PSA 的权限。 + - 此选项将引入安全风险，因为运营商现在将能够为其有权访问的命名空间设置 PSA。这可能允许操作员部署特权 Pod，或通过其他方式实现集群接管。 +- 具有适当权限的用户帐户可以使用适当的配置预先创建命名空间。 + - 此选项取决于应用程序处理现有资源的能力。 + +## 特定版本的问题 + +**注意：** 以下是影响特定 Rancher/webhook 版本的高严重性问题的不完整列表。在大多数情况下，这些问题可以通过升级到更新的 Rancher 版本来解决。 + +### 回滚到不兼容的 Webhook 版本 + +**注意：** 这会影响回滚到 Rancher v2.7.5 或更早版本。 + +如果回滚到 Rancher v2.7.5 或更早版本，您可能会看到 webhook 版本太新，无法与运行 v2.7.5 之前版本的 Rancher 的下游集群兼容。这可能会导致各种不兼容问题。例如，项目成员可能无法创建命名空间。此外，当您回滚到下游集群中安装 webhook 之前的版本时，webhook 可能仍保持安装状态，这会导致类似的不兼容问题。 + +为了帮助缓解这些问题，您可以在回滚后运行 [adjust-downstream-webhook](https://github.com/rancherlabs/support-tools/tree/master/adjust-downstream-webhook) shell 脚本。该脚本为相应的 Rancher 版本选择并安装正确的 webhook 版本（或完全删除 webhook）。 + +### 项目用户无法创建命名空间 + +**注意：** 以下内容影响 Rancher v2.7.2 - v2.7.4。 + +项目用户可能无法在项目中创建命名空间，这包括项目所有者。此问题是由于 Rancher 自动将 webhook 升级到与当前安装的 Rancher 版本更新的版本不兼容而导致的。 + +为了帮助缓解这些问题，您可以在回滚后运行 [adjust-downstream-webhook](https://github.com/rancherlabs/support-tools/tree/master/adjust-downstream-webhook) shell 脚本。该脚本为相应的 Rancher 版本选择并安装正确的 webhook 版本（或完全删除 webhook）。 From 6a52c6b462cd9a3d240cac6208f7ffa4e0275e6b Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Mon, 29 Apr 2024 16:48:22 -0700 Subject: [PATCH 58/67] Move cattle agent cpu and mem rec to cluster agent specific page. Revert additions to incorrect pages --- .../about-rancher-agents.md | 55 ++++++++++++++++++- .../tips-to-set-up-containers.md | 7 --- .../about-rancher-agents.md | 55 ++++++++++++++++++- .../tips-to-set-up-containers.md | 7 +-- .../about-rancher-agents.md | 55 ++++++++++++++++++- .../tips-to-set-up-containers.md | 7 +-- 6 files changed, 161 insertions(+), 25 deletions(-) diff --git a/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md b/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md index 2d98149dae2..57d3611f5b1 100644 --- a/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md +++ b/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md @@ -21,11 +21,62 @@ The `cattle-cluster-agent` is used to connect to the Kubernetes API of [Rancher The `cattle-node-agent` is used to interact with nodes in a [Rancher Launched Kubernetes](launch-kubernetes-with-rancher.md) cluster when performing cluster operations. Examples of cluster operations are upgrading Kubernetes version and creating/restoring etcd snapshots. The `cattle-node-agent` is deployed using a DaemonSet resource to make sure it runs on every node. The `cattle-node-agent` is used as fallback option to connect to the Kubernetes API of [Rancher Launched Kubernetes](launch-kubernetes-with-rancher.md) clusters when `cattle-cluster-agent` is unavailable. +### Requests + +The `cattle-cluster-agent` pod does not define default CPU and memory request values. As a baseline, we recommend setting the CPU request at `50m` and memory request at `100Mi`. However, it is important that you ensure your cluster has the correct resources available and the use case has been assessed appropriately before doing so. + +To configure request values through the UI: + + + + +1. When you create or edit an existing cluster, go to the **Cluster Options** section. +1. Expand the **Cluster Configuration** subsection. +1. Configure your request values using the **CPU Requests** and **Memory Requests** fields as needed. + + + + +1. When you create or edit an existing cluster, go to the **Cluster Configuration**. +1. Select the **Cluster Agent** subsection. +1. Configure your request values using the **CPU Reservation** and **Memory Reservation** fields as needed. + + + + +If you prefer to configure via YAML, add the following snippet to your configuration file: + + + + +```yaml +cluster_agent_deployment_customization: + override_resource_requirements: + requests: + cpu: 50m + memory: 100Mi +``` + + + + +```yaml +spec: + clusterAgentDeploymentCustomization: + overrideResourceRequirements: + requests: + cpu: 50m + memory: 100Mi +``` + + + + ### Scheduling rules -The `cattle-cluster-agent` uses either a fixed set of tolerations, or dynamically-added tolerations based on taints applied to the control plane nodes. This structure allows [Taint based Evictions](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions) to work properly for `cattle-cluster-agent`. +The `cattle-cluster-agent` uses either a fixed set of tolerations, or dynamically-added tolerations based on taints applied to the control plane nodes. This structure allows [Taint based Evictions](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions) to work properly for `cattle-cluster-agent`. -If control plane nodes are present in the cluster, the default tolerations will be replaced with tolerations matching the taints on the control plane nodes. The default set of tolerations are described below. +If control plane nodes are present in the cluster, the default tolerations will be replaced with tolerations matching the taints on the control plane nodes. The default set of tolerations are described below. | Component | nodeAffinity nodeSelectorTerms | nodeSelector | Tolerations | | ---------------------- | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------ | diff --git a/docs/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md b/docs/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md index 343cb9fd151..0cec0a91864 100644 --- a/docs/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md +++ b/docs/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md @@ -21,7 +21,6 @@ Smaller distributions such as Alpine and BusyBox reduce container image size and Popular distributions such as Ubuntu, Fedora, and CentOS are more field-tested and offer more functionality. ### Start with a FROM scratch container - If your microservice is a standalone static binary, you should use a FROM scratch container. The FROM scratch container is an [official Docker image](https://hub.docker.com/_/scratch) that is empty so that you can use it to design minimal images. @@ -29,11 +28,9 @@ The FROM scratch container is an [official Docker image](https://hub.docker.com/ This will have the smallest attack surface and smallest image size. ### Run Container Processes as Unprivileged - When possible, use a non-privileged user when running processes within your container. While container runtimes provide isolation, vulnerabilities and attacks are still possible. Inadvertent or accidental host mounts can also be impacted if the container is running as root. For details on configuring a security context for a pod or container, refer to the [Kubernetes docs](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). ### Define Resource Limits - Apply CPU and memory limits to your pods. This can help manage the resources on your worker nodes and avoid a malfunctioning microservice from impacting other microservices. In standard Kubernetes, you can set resource limits on the namespace level. In Rancher, you can set resource limits on the project level and they will propagate to all the namespaces within the project. For details, refer to the Rancher docs. @@ -43,7 +40,6 @@ When setting resource quotas, if you set anything related to CPU or Memory (i.e. The Kubernetes docs have more information on how resource limits can be set at the [container level](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) and the namespace level. ### Define Resource Requirements - You should apply CPU and memory requirements to your pods. This is crucial for informing the scheduler which type of compute node your pod needs to be placed on, and ensuring it does not over-provision that node. In Kubernetes, you can set a resource requirement by defining `resources.requests` in the resource requests field in a pod's container spec. For details, refer to the [Kubernetes docs](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container). :::note @@ -54,10 +50,7 @@ If you set a resource limit for the namespace that the pod is deployed in, and t It is recommended to define resource requirements on the container level because otherwise, the scheduler makes assumptions that will likely not be helpful to your application when the cluster experiences load. -TODO we recommend to set your CPU request at 50m and memory request at 100Mi, however this is a base recommendation and customers should choose based on resources available in their environment and their use case on how they use Rancher. - ### Liveness and Readiness Probes - Set up liveness and readiness probes for your container. Unless your container completely crashes, Kubernetes will not know it's unhealthy unless you create an endpoint or mechanism that can report container status. Alternatively, make sure your container halts and crashes if unhealthy. The Kubernetes docs show how to [configure liveness and readiness probes for containers.](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md index 2d98149dae2..57d3611f5b1 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md @@ -21,11 +21,62 @@ The `cattle-cluster-agent` is used to connect to the Kubernetes API of [Rancher The `cattle-node-agent` is used to interact with nodes in a [Rancher Launched Kubernetes](launch-kubernetes-with-rancher.md) cluster when performing cluster operations. Examples of cluster operations are upgrading Kubernetes version and creating/restoring etcd snapshots. The `cattle-node-agent` is deployed using a DaemonSet resource to make sure it runs on every node. The `cattle-node-agent` is used as fallback option to connect to the Kubernetes API of [Rancher Launched Kubernetes](launch-kubernetes-with-rancher.md) clusters when `cattle-cluster-agent` is unavailable. +### Requests + +The `cattle-cluster-agent` pod does not define default CPU and memory request values. As a baseline, we recommend setting the CPU request at `50m` and memory request at `100Mi`. However, it is important that you ensure your cluster has the correct resources available and the use case has been assessed appropriately before doing so. + +To configure request values through the UI: + + + + +1. When you create or edit an existing cluster, go to the **Cluster Options** section. +1. Expand the **Cluster Configuration** subsection. +1. Configure your request values using the **CPU Requests** and **Memory Requests** fields as needed. + + + + +1. When you create or edit an existing cluster, go to the **Cluster Configuration**. +1. Select the **Cluster Agent** subsection. +1. Configure your request values using the **CPU Reservation** and **Memory Reservation** fields as needed. + + + + +If you prefer to configure via YAML, add the following snippet to your configuration file: + + + + +```yaml +cluster_agent_deployment_customization: + override_resource_requirements: + requests: + cpu: 50m + memory: 100Mi +``` + + + + +```yaml +spec: + clusterAgentDeploymentCustomization: + overrideResourceRequirements: + requests: + cpu: 50m + memory: 100Mi +``` + + + + ### Scheduling rules -The `cattle-cluster-agent` uses either a fixed set of tolerations, or dynamically-added tolerations based on taints applied to the control plane nodes. This structure allows [Taint based Evictions](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions) to work properly for `cattle-cluster-agent`. +The `cattle-cluster-agent` uses either a fixed set of tolerations, or dynamically-added tolerations based on taints applied to the control plane nodes. This structure allows [Taint based Evictions](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions) to work properly for `cattle-cluster-agent`. -If control plane nodes are present in the cluster, the default tolerations will be replaced with tolerations matching the taints on the control plane nodes. The default set of tolerations are described below. +If control plane nodes are present in the cluster, the default tolerations will be replaced with tolerations matching the taints on the control plane nodes. The default set of tolerations are described below. | Component | nodeAffinity nodeSelectorTerms | nodeSelector | Tolerations | | ---------------------- | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------ | diff --git a/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md b/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md index 98dfae1ffaa..0cec0a91864 100644 --- a/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md +++ b/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md @@ -21,7 +21,6 @@ Smaller distributions such as Alpine and BusyBox reduce container image size and Popular distributions such as Ubuntu, Fedora, and CentOS are more field-tested and offer more functionality. ### Start with a FROM scratch container - If your microservice is a standalone static binary, you should use a FROM scratch container. The FROM scratch container is an [official Docker image](https://hub.docker.com/_/scratch) that is empty so that you can use it to design minimal images. @@ -29,11 +28,9 @@ The FROM scratch container is an [official Docker image](https://hub.docker.com/ This will have the smallest attack surface and smallest image size. ### Run Container Processes as Unprivileged - When possible, use a non-privileged user when running processes within your container. While container runtimes provide isolation, vulnerabilities and attacks are still possible. Inadvertent or accidental host mounts can also be impacted if the container is running as root. For details on configuring a security context for a pod or container, refer to the [Kubernetes docs](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). ### Define Resource Limits - Apply CPU and memory limits to your pods. This can help manage the resources on your worker nodes and avoid a malfunctioning microservice from impacting other microservices. In standard Kubernetes, you can set resource limits on the namespace level. In Rancher, you can set resource limits on the project level and they will propagate to all the namespaces within the project. For details, refer to the Rancher docs. @@ -43,7 +40,6 @@ When setting resource quotas, if you set anything related to CPU or Memory (i.e. The Kubernetes docs have more information on how resource limits can be set at the [container level](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) and the namespace level. ### Define Resource Requirements - You should apply CPU and memory requirements to your pods. This is crucial for informing the scheduler which type of compute node your pod needs to be placed on, and ensuring it does not over-provision that node. In Kubernetes, you can set a resource requirement by defining `resources.requests` in the resource requests field in a pod's container spec. For details, refer to the [Kubernetes docs](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container). :::note @@ -52,10 +48,9 @@ If you set a resource limit for the namespace that the pod is deployed in, and t ::: -As a starting point, we recommend setting your CPU request at 50m and memory request at 100Mi. You should choose your limits based on the resources available in your environment and your use case with Rancher. It is recommended to define resource requirements on the container level because otherwise, the scheduler makes assumptions that will likely not be helpful to your application when the cluster experiences load. +It is recommended to define resource requirements on the container level because otherwise, the scheduler makes assumptions that will likely not be helpful to your application when the cluster experiences load. ### Liveness and Readiness Probes - Set up liveness and readiness probes for your container. Unless your container completely crashes, Kubernetes will not know it's unhealthy unless you create an endpoint or mechanism that can report container status. Alternatively, make sure your container halts and crashes if unhealthy. The Kubernetes docs show how to [configure liveness and readiness probes for containers.](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md index 2d98149dae2..57d3611f5b1 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md @@ -21,11 +21,62 @@ The `cattle-cluster-agent` is used to connect to the Kubernetes API of [Rancher The `cattle-node-agent` is used to interact with nodes in a [Rancher Launched Kubernetes](launch-kubernetes-with-rancher.md) cluster when performing cluster operations. Examples of cluster operations are upgrading Kubernetes version and creating/restoring etcd snapshots. The `cattle-node-agent` is deployed using a DaemonSet resource to make sure it runs on every node. The `cattle-node-agent` is used as fallback option to connect to the Kubernetes API of [Rancher Launched Kubernetes](launch-kubernetes-with-rancher.md) clusters when `cattle-cluster-agent` is unavailable. +### Requests + +The `cattle-cluster-agent` pod does not define default CPU and memory request values. As a baseline, we recommend setting the CPU request at `50m` and memory request at `100Mi`. However, it is important that you ensure your cluster has the correct resources available and the use case has been assessed appropriately before doing so. + +To configure request values through the UI: + + + + +1. When you create or edit an existing cluster, go to the **Cluster Options** section. +1. Expand the **Cluster Configuration** subsection. +1. Configure your request values using the **CPU Requests** and **Memory Requests** fields as needed. + + + + +1. When you create or edit an existing cluster, go to the **Cluster Configuration**. +1. Select the **Cluster Agent** subsection. +1. Configure your request values using the **CPU Reservation** and **Memory Reservation** fields as needed. + + + + +If you prefer to configure via YAML, add the following snippet to your configuration file: + + + + +```yaml +cluster_agent_deployment_customization: + override_resource_requirements: + requests: + cpu: 50m + memory: 100Mi +``` + + + + +```yaml +spec: + clusterAgentDeploymentCustomization: + overrideResourceRequirements: + requests: + cpu: 50m + memory: 100Mi +``` + + + + ### Scheduling rules -The `cattle-cluster-agent` uses either a fixed set of tolerations, or dynamically-added tolerations based on taints applied to the control plane nodes. This structure allows [Taint based Evictions](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions) to work properly for `cattle-cluster-agent`. +The `cattle-cluster-agent` uses either a fixed set of tolerations, or dynamically-added tolerations based on taints applied to the control plane nodes. This structure allows [Taint based Evictions](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#taint-based-evictions) to work properly for `cattle-cluster-agent`. -If control plane nodes are present in the cluster, the default tolerations will be replaced with tolerations matching the taints on the control plane nodes. The default set of tolerations are described below. +If control plane nodes are present in the cluster, the default tolerations will be replaced with tolerations matching the taints on the control plane nodes. The default set of tolerations are described below. | Component | nodeAffinity nodeSelectorTerms | nodeSelector | Tolerations | | ---------------------- | ------------------------------------------ | ------------ | ------------------------------------------------------------------------------ | diff --git a/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md b/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md index 98dfae1ffaa..0cec0a91864 100644 --- a/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md +++ b/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/tips-to-set-up-containers.md @@ -21,7 +21,6 @@ Smaller distributions such as Alpine and BusyBox reduce container image size and Popular distributions such as Ubuntu, Fedora, and CentOS are more field-tested and offer more functionality. ### Start with a FROM scratch container - If your microservice is a standalone static binary, you should use a FROM scratch container. The FROM scratch container is an [official Docker image](https://hub.docker.com/_/scratch) that is empty so that you can use it to design minimal images. @@ -29,11 +28,9 @@ The FROM scratch container is an [official Docker image](https://hub.docker.com/ This will have the smallest attack surface and smallest image size. ### Run Container Processes as Unprivileged - When possible, use a non-privileged user when running processes within your container. While container runtimes provide isolation, vulnerabilities and attacks are still possible. Inadvertent or accidental host mounts can also be impacted if the container is running as root. For details on configuring a security context for a pod or container, refer to the [Kubernetes docs](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/). ### Define Resource Limits - Apply CPU and memory limits to your pods. This can help manage the resources on your worker nodes and avoid a malfunctioning microservice from impacting other microservices. In standard Kubernetes, you can set resource limits on the namespace level. In Rancher, you can set resource limits on the project level and they will propagate to all the namespaces within the project. For details, refer to the Rancher docs. @@ -43,7 +40,6 @@ When setting resource quotas, if you set anything related to CPU or Memory (i.e. The Kubernetes docs have more information on how resource limits can be set at the [container level](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) and the namespace level. ### Define Resource Requirements - You should apply CPU and memory requirements to your pods. This is crucial for informing the scheduler which type of compute node your pod needs to be placed on, and ensuring it does not over-provision that node. In Kubernetes, you can set a resource requirement by defining `resources.requests` in the resource requests field in a pod's container spec. For details, refer to the [Kubernetes docs](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container). :::note @@ -52,10 +48,9 @@ If you set a resource limit for the namespace that the pod is deployed in, and t ::: -As a starting point, we recommend setting your CPU request at 50m and memory request at 100Mi. You should choose your limits based on the resources available in your environment and your use case with Rancher. It is recommended to define resource requirements on the container level because otherwise, the scheduler makes assumptions that will likely not be helpful to your application when the cluster experiences load. +It is recommended to define resource requirements on the container level because otherwise, the scheduler makes assumptions that will likely not be helpful to your application when the cluster experiences load. ### Liveness and Readiness Probes - Set up liveness and readiness probes for your container. Unless your container completely crashes, Kubernetes will not know it's unhealthy unless you create an endpoint or mechanism that can report container status. Alternatively, make sure your container halts and crashes if unhealthy. The Kubernetes docs show how to [configure liveness and readiness probes for containers.](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/) From 8b71096b32e125601735361546d2781c552d0d28 Mon Sep 17 00:00:00 2001 From: Billy Tat Date: Tue, 30 Apr 2024 11:04:41 -0700 Subject: [PATCH 59/67] Apply feedback --- .../launch-kubernetes-with-rancher/about-rancher-agents.md | 6 +++--- .../launch-kubernetes-with-rancher/about-rancher-agents.md | 6 +++--- .../launch-kubernetes-with-rancher/about-rancher-agents.md | 6 +++--- 3 files changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md b/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md index 57d3611f5b1..375397eb513 100644 --- a/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md +++ b/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md @@ -23,21 +23,21 @@ The `cattle-node-agent` is used to interact with nodes in a [Rancher Launched Ku ### Requests -The `cattle-cluster-agent` pod does not define default CPU and memory request values. As a baseline, we recommend setting the CPU request at `50m` and memory request at `100Mi`. However, it is important that you ensure your cluster has the correct resources available and the use case has been assessed appropriately before doing so. +The `cattle-cluster-agent` pod does not define the default CPU and memory request values. As a baseline, we recommend setting the CPU request at `50m` and memory request at `100Mi`. However, it is important that you assess your use case appropriately and that you allocate the correct resources to your cluster for your needs. To configure request values through the UI: -1. When you create or edit an existing cluster, go to the **Cluster Options** section. +1. When you [create](./launch-kubernetes-with-rancher.md) or edit an existing cluster, go to the **Cluster Options** section. 1. Expand the **Cluster Configuration** subsection. 1. Configure your request values using the **CPU Requests** and **Memory Requests** fields as needed. -1. When you create or edit an existing cluster, go to the **Cluster Configuration**. +1. When you [create](./launch-kubernetes-with-rancher.md) or edit an existing cluster, go to the **Cluster Configuration**. 1. Select the **Cluster Agent** subsection. 1. Configure your request values using the **CPU Reservation** and **Memory Reservation** fields as needed. diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md index 57d3611f5b1..375397eb513 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md @@ -23,21 +23,21 @@ The `cattle-node-agent` is used to interact with nodes in a [Rancher Launched Ku ### Requests -The `cattle-cluster-agent` pod does not define default CPU and memory request values. As a baseline, we recommend setting the CPU request at `50m` and memory request at `100Mi`. However, it is important that you ensure your cluster has the correct resources available and the use case has been assessed appropriately before doing so. +The `cattle-cluster-agent` pod does not define the default CPU and memory request values. As a baseline, we recommend setting the CPU request at `50m` and memory request at `100Mi`. However, it is important that you assess your use case appropriately and that you allocate the correct resources to your cluster for your needs. To configure request values through the UI: -1. When you create or edit an existing cluster, go to the **Cluster Options** section. +1. When you [create](./launch-kubernetes-with-rancher.md) or edit an existing cluster, go to the **Cluster Options** section. 1. Expand the **Cluster Configuration** subsection. 1. Configure your request values using the **CPU Requests** and **Memory Requests** fields as needed. -1. When you create or edit an existing cluster, go to the **Cluster Configuration**. +1. When you [create](./launch-kubernetes-with-rancher.md) or edit an existing cluster, go to the **Cluster Configuration**. 1. Select the **Cluster Agent** subsection. 1. Configure your request values using the **CPU Reservation** and **Memory Reservation** fields as needed. diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md index 57d3611f5b1..375397eb513 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/about-rancher-agents.md @@ -23,21 +23,21 @@ The `cattle-node-agent` is used to interact with nodes in a [Rancher Launched Ku ### Requests -The `cattle-cluster-agent` pod does not define default CPU and memory request values. As a baseline, we recommend setting the CPU request at `50m` and memory request at `100Mi`. However, it is important that you ensure your cluster has the correct resources available and the use case has been assessed appropriately before doing so. +The `cattle-cluster-agent` pod does not define the default CPU and memory request values. As a baseline, we recommend setting the CPU request at `50m` and memory request at `100Mi`. However, it is important that you assess your use case appropriately and that you allocate the correct resources to your cluster for your needs. To configure request values through the UI: -1. When you create or edit an existing cluster, go to the **Cluster Options** section. +1. When you [create](./launch-kubernetes-with-rancher.md) or edit an existing cluster, go to the **Cluster Options** section. 1. Expand the **Cluster Configuration** subsection. 1. Configure your request values using the **CPU Requests** and **Memory Requests** fields as needed. -1. When you create or edit an existing cluster, go to the **Cluster Configuration**. +1. When you [create](./launch-kubernetes-with-rancher.md) or edit an existing cluster, go to the **Cluster Configuration**. 1. Select the **Cluster Agent** subsection. 1. Configure your request values using the **CPU Reservation** and **Memory Reservation** fields as needed. From 54dc6b187bb72ca0cd573cfbf6b4f71815496c4c Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Tue, 30 Apr 2024 17:21:19 -0400 Subject: [PATCH 60/67] #1228 Overlay test for Windows nodes (#1239) * 1228 Overlay test for Windows nodes * Apply suggestions from code review Co-authored-by: Sunil Singh * updated note about error message * versioned note on 2.0-2.4 --------- Co-authored-by: Sunil Singh --- .../troubleshooting/other-troubleshooting-tips/networking.md | 5 +++-- .../troubleshooting/other-troubleshooting-tips/networking.md | 2 +- .../troubleshooting/other-troubleshooting-tips/networking.md | 2 +- .../troubleshooting/other-troubleshooting-tips/networking.md | 2 +- .../troubleshooting/other-troubleshooting-tips/networking.md | 2 +- .../troubleshooting/other-troubleshooting-tips/networking.md | 2 +- 6 files changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/troubleshooting/other-troubleshooting-tips/networking.md b/docs/troubleshooting/other-troubleshooting-tips/networking.md index bea850ffe13..d67a1cdb793 100644 --- a/docs/troubleshooting/other-troubleshooting-tips/networking.md +++ b/docs/troubleshooting/other-troubleshooting-tips/networking.md @@ -13,15 +13,16 @@ Make sure you configured the correct kubeconfig (for example, `export KUBECONFIG ### Double check if all the required ports are opened in your (host) firewall Double check if all the [required ports](../../how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md#networking-requirements) are opened in your (host) firewall. The overlay network uses UDP in comparison to all other required ports which are TCP. + ### Check if overlay network is functioning correctly The pod can be scheduled to any of the hosts you used for your cluster, but that means that the NGINX ingress controller needs to be able to route the request from `NODE_1` to `NODE_2`. This happens over the overlay network. If the overlay network is not functioning, you will experience intermittent TCP/HTTP connection failures due to the NGINX ingress controller not being able to route to the pod. To test the overlay network, you can launch the following `DaemonSet` definition. This will run a `swiss-army-knife` container on every host (image was developed by Rancher engineers and can be found here: https://github.com/rancherlabs/swiss-army-knife), which we will use to run a `ping` test between containers on all hosts. -:::note +:::caution -This container [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. This will be seen in the pod logs as `exec user process caused: exec format error`. +The `swiss-army-knife` container does not support Windows nodes. It also [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. When the test encounters incompatible nodes, this is recorded in the pod logs as an error message, such as `exec user process caused: exec format error` for ARM nodes, or `ImagePullBackOff (Back-off pulling image "rancherlabs/swiss-army-knife)` for Windows nodes. ::: diff --git a/versioned_docs/version-2.0-2.4/troubleshooting/other-troubleshooting-tips/networking.md b/versioned_docs/version-2.0-2.4/troubleshooting/other-troubleshooting-tips/networking.md index f1bffa50c6d..e75a9783b97 100644 --- a/versioned_docs/version-2.0-2.4/troubleshooting/other-troubleshooting-tips/networking.md +++ b/versioned_docs/version-2.0-2.4/troubleshooting/other-troubleshooting-tips/networking.md @@ -19,7 +19,7 @@ The pod can be scheduled to any of the hosts you used for your cluster, but that To test the overlay network, you can launch the following `DaemonSet` definition. This will run a `swiss-army-knife` container on every host (image was developed by Rancher engineers and can be found here: https://github.com/rancherlabs/swiss-army-knife), which we will use to run a `ping` test between containers on all hosts. -> **Note:** This container [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. This will be seen in the pod logs as `exec user process caused: exec format error`. +> **Note:** The `swiss-army-knife` container does not support Windows nodes. It also [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. When the test encounters incompatible nodes, this is recorded in the pod logs as an error message, such as `exec user process caused: exec format error` for ARM nodes, or `ImagePullBackOff (Back-off pulling image "rancherlabs/swiss-army-knife)` for Windows nodes. 1. Save the following file as `overlaytest.yml` diff --git a/versioned_docs/version-2.5/troubleshooting/other-troubleshooting-tips/networking.md b/versioned_docs/version-2.5/troubleshooting/other-troubleshooting-tips/networking.md index 9427ae11f9e..e1d6c78cfe5 100644 --- a/versioned_docs/version-2.5/troubleshooting/other-troubleshooting-tips/networking.md +++ b/versioned_docs/version-2.5/troubleshooting/other-troubleshooting-tips/networking.md @@ -19,7 +19,7 @@ The pod can be scheduled to any of the hosts you used for your cluster, but that To test the overlay network, you can launch the following `DaemonSet` definition. This will run a `swiss-army-knife` container on every host (image was developed by Rancher engineers and can be found here: https://github.com/rancherlabs/swiss-army-knife), which we will use to run a `ping` test between containers on all hosts. -> **Note:** This container [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. This will be seen in the pod logs as `exec user process caused: exec format error`. +> **Note:** The `swiss-army-knife` container does not support Windows nodes. It also [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. When the test encounters incompatible nodes, this is recorded in the pod logs as an error message, such as `exec user process caused: exec format error` for ARM nodes, or `ImagePullBackOff (Back-off pulling image "rancherlabs/swiss-army-knife)` for Windows nodes. 1. Save the following file as `overlaytest.yml` diff --git a/versioned_docs/version-2.6/troubleshooting/other-troubleshooting-tips/networking.md b/versioned_docs/version-2.6/troubleshooting/other-troubleshooting-tips/networking.md index bea850ffe13..9c95ff1d0e5 100644 --- a/versioned_docs/version-2.6/troubleshooting/other-troubleshooting-tips/networking.md +++ b/versioned_docs/version-2.6/troubleshooting/other-troubleshooting-tips/networking.md @@ -21,7 +21,7 @@ To test the overlay network, you can launch the following `DaemonSet` definition :::note -This container [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. This will be seen in the pod logs as `exec user process caused: exec format error`. +The `swiss-army-knife` container does not support Windows nodes. It also [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. When the test encounters incompatible nodes, this is recorded in the pod logs as an error message, such as `exec user process caused: exec format error` for ARM nodes, or `ImagePullBackOff (Back-off pulling image "rancherlabs/swiss-army-knife)` for Windows nodes. ::: diff --git a/versioned_docs/version-2.7/troubleshooting/other-troubleshooting-tips/networking.md b/versioned_docs/version-2.7/troubleshooting/other-troubleshooting-tips/networking.md index bea850ffe13..9c95ff1d0e5 100644 --- a/versioned_docs/version-2.7/troubleshooting/other-troubleshooting-tips/networking.md +++ b/versioned_docs/version-2.7/troubleshooting/other-troubleshooting-tips/networking.md @@ -21,7 +21,7 @@ To test the overlay network, you can launch the following `DaemonSet` definition :::note -This container [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. This will be seen in the pod logs as `exec user process caused: exec format error`. +The `swiss-army-knife` container does not support Windows nodes. It also [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. When the test encounters incompatible nodes, this is recorded in the pod logs as an error message, such as `exec user process caused: exec format error` for ARM nodes, or `ImagePullBackOff (Back-off pulling image "rancherlabs/swiss-army-knife)` for Windows nodes. ::: diff --git a/versioned_docs/version-2.8/troubleshooting/other-troubleshooting-tips/networking.md b/versioned_docs/version-2.8/troubleshooting/other-troubleshooting-tips/networking.md index bea850ffe13..9c95ff1d0e5 100644 --- a/versioned_docs/version-2.8/troubleshooting/other-troubleshooting-tips/networking.md +++ b/versioned_docs/version-2.8/troubleshooting/other-troubleshooting-tips/networking.md @@ -21,7 +21,7 @@ To test the overlay network, you can launch the following `DaemonSet` definition :::note -This container [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. This will be seen in the pod logs as `exec user process caused: exec format error`. +The `swiss-army-knife` container does not support Windows nodes. It also [does not support ARM nodes](https://github.com/leodotcloud/swiss-army-knife/issues/18), such as a Raspberry Pi. When the test encounters incompatible nodes, this is recorded in the pod logs as an error message, such as `exec user process caused: exec format error` for ARM nodes, or `ImagePullBackOff (Back-off pulling image "rancherlabs/swiss-army-knife)` for Windows nodes. ::: From 32d05e9489924e53c6f64d5ea1d2f2ddf4ea3288 Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Wed, 1 May 2024 13:10:23 -0400 Subject: [PATCH 61/67] Added note about cve-2024-22030 to security faq (#1244) * added note about cve-2024-22030 to security faq * Apply suggestions from code review Co-authored-by: Sunil Singh * Apply suggestions from code review Co-authored-by: Billy Tat * suggestions from Slack applied * versioning --------- Co-authored-by: Sunil Singh Co-authored-by: Billy Tat --- docs/faq/security.md | 14 +++++++++----- versioned_docs/version-2.7/faq/security.md | 14 +++++++++----- versioned_docs/version-2.8/faq/security.md | 14 +++++++++----- 3 files changed, 27 insertions(+), 15 deletions(-) diff --git a/docs/faq/security.md b/docs/faq/security.md index 46f774bf722..08fd8422730 100644 --- a/docs/faq/security.md +++ b/docs/faq/security.md @@ -7,12 +7,16 @@ title: Security FAQ -**Is there a Hardening Guide?** +### Is there a Hardening Guide? -The Hardening Guide is now located in the main [Security](../reference-guides/rancher-security/rancher-security.md) section. +The Hardening Guide is located in the main [Security](../reference-guides/rancher-security/rancher-security.md) section. -
- -**What are the results of Rancher's Kubernetes cluster when it is CIS benchmarked?** +### Have hardened Rancher Kubernetes clusters been evaluated by the CIS Kubernetes Benchmark? Where can I find the results? We have run the CIS Kubernetes benchmark against a hardened Rancher Kubernetes cluster. The results of that assessment can be found in the main [Security](../reference-guides/rancher-security/rancher-security.md) section. + +### How does Rancher verify communication with downstream clusters, and what are some associated security concerns? + +Communication between the Rancher server and downstream clusters is performed through agents. Rancher uses either a registered certificate authority (CA) bundle or the local trust store to verify communication between Rancher agents and the Rancher server. Using a CA bundle for verification is more strict, as only the certificates based on that bundle are trusted. If TLS verification for a explicit CA bundle fails, Rancher may fall back to using the local trust store for verifying future communication. Any CA within the local trust store can then be used to generate a valid certificate. + +As described in [Rancher Security Update CVE-2024-22030](https://www.suse.com/c/rancher-security-update/), under a narrow set of circumstances, malicious actors can take over Rancher nodes by exploiting the behavior of Rancher CAs. For the attack to succeed, the malicious actor must generate a valid certificate from either a valid CA in the targeted Rancher server, or from a valid registered CA. The attacker also needs to either hijack or spoof the Rancher server-url as a preliminary step. Rancher is currently evaluating Rancher CA behavior to mitigate against this and any similar avenues of attack. diff --git a/versioned_docs/version-2.7/faq/security.md b/versioned_docs/version-2.7/faq/security.md index 46f774bf722..08fd8422730 100644 --- a/versioned_docs/version-2.7/faq/security.md +++ b/versioned_docs/version-2.7/faq/security.md @@ -7,12 +7,16 @@ title: Security FAQ -**Is there a Hardening Guide?** +### Is there a Hardening Guide? -The Hardening Guide is now located in the main [Security](../reference-guides/rancher-security/rancher-security.md) section. +The Hardening Guide is located in the main [Security](../reference-guides/rancher-security/rancher-security.md) section. -
- -**What are the results of Rancher's Kubernetes cluster when it is CIS benchmarked?** +### Have hardened Rancher Kubernetes clusters been evaluated by the CIS Kubernetes Benchmark? Where can I find the results? We have run the CIS Kubernetes benchmark against a hardened Rancher Kubernetes cluster. The results of that assessment can be found in the main [Security](../reference-guides/rancher-security/rancher-security.md) section. + +### How does Rancher verify communication with downstream clusters, and what are some associated security concerns? + +Communication between the Rancher server and downstream clusters is performed through agents. Rancher uses either a registered certificate authority (CA) bundle or the local trust store to verify communication between Rancher agents and the Rancher server. Using a CA bundle for verification is more strict, as only the certificates based on that bundle are trusted. If TLS verification for a explicit CA bundle fails, Rancher may fall back to using the local trust store for verifying future communication. Any CA within the local trust store can then be used to generate a valid certificate. + +As described in [Rancher Security Update CVE-2024-22030](https://www.suse.com/c/rancher-security-update/), under a narrow set of circumstances, malicious actors can take over Rancher nodes by exploiting the behavior of Rancher CAs. For the attack to succeed, the malicious actor must generate a valid certificate from either a valid CA in the targeted Rancher server, or from a valid registered CA. The attacker also needs to either hijack or spoof the Rancher server-url as a preliminary step. Rancher is currently evaluating Rancher CA behavior to mitigate against this and any similar avenues of attack. diff --git a/versioned_docs/version-2.8/faq/security.md b/versioned_docs/version-2.8/faq/security.md index 46f774bf722..08fd8422730 100644 --- a/versioned_docs/version-2.8/faq/security.md +++ b/versioned_docs/version-2.8/faq/security.md @@ -7,12 +7,16 @@ title: Security FAQ -**Is there a Hardening Guide?** +### Is there a Hardening Guide? -The Hardening Guide is now located in the main [Security](../reference-guides/rancher-security/rancher-security.md) section. +The Hardening Guide is located in the main [Security](../reference-guides/rancher-security/rancher-security.md) section. -
- -**What are the results of Rancher's Kubernetes cluster when it is CIS benchmarked?** +### Have hardened Rancher Kubernetes clusters been evaluated by the CIS Kubernetes Benchmark? Where can I find the results? We have run the CIS Kubernetes benchmark against a hardened Rancher Kubernetes cluster. The results of that assessment can be found in the main [Security](../reference-guides/rancher-security/rancher-security.md) section. + +### How does Rancher verify communication with downstream clusters, and what are some associated security concerns? + +Communication between the Rancher server and downstream clusters is performed through agents. Rancher uses either a registered certificate authority (CA) bundle or the local trust store to verify communication between Rancher agents and the Rancher server. Using a CA bundle for verification is more strict, as only the certificates based on that bundle are trusted. If TLS verification for a explicit CA bundle fails, Rancher may fall back to using the local trust store for verifying future communication. Any CA within the local trust store can then be used to generate a valid certificate. + +As described in [Rancher Security Update CVE-2024-22030](https://www.suse.com/c/rancher-security-update/), under a narrow set of circumstances, malicious actors can take over Rancher nodes by exploiting the behavior of Rancher CAs. For the attack to succeed, the malicious actor must generate a valid certificate from either a valid CA in the targeted Rancher server, or from a valid registered CA. The attacker also needs to either hijack or spoof the Rancher server-url as a preliminary step. Rancher is currently evaluating Rancher CA behavior to mitigate against this and any similar avenues of attack. From 9ea762eb4efdd4c02198243263ee38f986a81d7d Mon Sep 17 00:00:00 2001 From: Lucas Saintarbor Date: Wed, 1 May 2024 14:38:34 -0700 Subject: [PATCH 62/67] Add Vale GitHub workflow (#1196) * Add Vale config file * Add GH workflow * Add SUSE style guide rules * Add reference of SUSE style guide and Vale to README Co-authored-by: Billy Tat --------- Co-authored-by: Billy Tat --- .github/styles/suse-vale-styleguide | 1 + .github/workflows/vale.yml | 53 +++++++++++++++++++++++++++++ .gitmodules | 3 ++ .vale.ini | 7 ++++ README.md | 12 ++++++- 5 files changed, 75 insertions(+), 1 deletion(-) create mode 160000 .github/styles/suse-vale-styleguide create mode 100644 .github/workflows/vale.yml create mode 100644 .gitmodules create mode 100644 .vale.ini diff --git a/.github/styles/suse-vale-styleguide b/.github/styles/suse-vale-styleguide new file mode 160000 index 00000000000..06f144fdfc7 --- /dev/null +++ b/.github/styles/suse-vale-styleguide @@ -0,0 +1 @@ +Subproject commit 06f144fdfc78c769a9d86db5a6c550d7e6fc17da diff --git a/.github/workflows/vale.yml b/.github/workflows/vale.yml new file mode 100644 index 00000000000..a695beac412 --- /dev/null +++ b/.github/workflows/vale.yml @@ -0,0 +1,53 @@ +# This action gets all changed markdown files in /docs and /versioned_docs using tj-actions/changed-files@v42 +# It compares new commits in the PR with the base commit (github.event.pull_request.base.sha) +# It checks if no markdown files are changed +# It shows a count of markdown files and lists all changed markdown files +# It uses Vale (https://vale.sh/docs/vale-cli/installation/) to provide feedback base off the SUSE Style Guide / OpenSUSE style rules (https://github.com/openSUSE/suse-vale-styleguide) + +name: Style check +on: [pull_request] + +jobs: + vale-lint: + name: runner / vale + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + with: + fetch-depth: 0 # OR "2" -> To retrieve the preceding commit. + submodules: true + - name: Get all changed markdown files + id: changed-markdown-files + uses: tj-actions/changed-files@v42 + with: + # Avoid using single or double quotes for multiline patterns + files: | + docs/** + versioned_docs/** + separator: "," + base_sha: ${{ github.event.pull_request.base.sha }} + env: + ALL_CHANGED_FILES: ${{ 0 }} + - name: No files changed? + if: steps.changed-markdown-files.outputs.any_changed == 'false' + run: | + echo "No files changed" + echo "ALL_CHANGED_FILES=$ALL_CHANGED_FILES" >> $GITHUB_ENV + - name: List all changed files markdown files + if: steps.changed-markdown-files.outputs.any_changed == 'true' + env: + ALL_CHANGED_FILES: ${{ steps.changed-markdown-files.outputs.all_changed_files }} + ALL_CHANGED_FILES_COUNT: ${{ steps.changed-markdown-files.outputs.all_changed_files_count }} + SHA: ${{ github.head_ref }} + HEAD: ${{ github.base_ref }} + run: | + echo "Total Files Changed:" ${ALL_CHANGED_FILES_COUNT} + echo ${ALL_CHANGED_FILES} + echo "ALL_CHANGED_FILES=$ALL_CHANGED_FILES" >> $GITHUB_ENV + - uses: errata-ai/vale-action@v2.1.0 + if: steps.changed-markdown-files.outputs.any_changed == 'true' + env: + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + with: + separator: ", " + files: ${{ env.ALL_CHANGED_FILES }} \ No newline at end of file diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 00000000000..1af7b495ba5 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule ".github/styles/suse-vale-styleguide"] + path = .github/styles/suse-vale-styleguide + url = https://github.com/openSUSE/suse-vale-styleguide diff --git a/.vale.ini b/.vale.ini new file mode 100644 index 00000000000..84cf4b55591 --- /dev/null +++ b/.vale.ini @@ -0,0 +1,7 @@ +StylesPath = .github/styles + +[formtats] +mdx = md + +[*.md] +BasedOnStyles = suse-vale-styleguide \ No newline at end of file diff --git a/README.md b/README.md index 6e9adbc8dac..ca3db799c77 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,17 @@ The file paths in the repo correspond to the URLs for pages on the docs website. ### Style & Formatting -The docs are written in [Markdown](https://www.markdownguide.org/getting-started/). We refer to the Microsoft [style guide](https://learn.microsoft.com/en-us/style-guide/welcome/) and use standard American English. Many pages are also available in Simplified Chinese. +The docs are written in [Markdown](https://www.markdownguide.org/getting-started/). We use standard American English and many pages are also available in Simplified Chinese. + +Moving forward, we are referring to the SUSE [style guide](https://documentation.suse.com/style/current/pdf/style-guide_en.pdf). The **Style check / runner / vale (pull_request)** check used [Vale](https://vale.sh/) to make style and grammar suggestions for new or updated documentation based on the SUSE style guide. To review these suggestions when working on a PR: + +1. Select the details of the **Style check / runner / vale (pull_request)** check. +1. In the logs, go to **Run errata-ai/vale-action@v2.1.0** and select **Running vale with reviewdog 🐶 ...** to view the suggestions. +1. New or updated files are checked against the SUSE style guide. Suggestions have the following format: '{"message": "[suse-vale-styleguide.Rule] Rule description", "location": {"path": "file-path", "range": {"start": {"line": , "column": }}}, "severity": " "}' + + For example: '{"message": "[suse-vale-styleguide.Usage] Use 'certain' instead of 'some'", "location": {"path": "docs/contribute-to-rancher.md", "range": {"start": {"line": 3, "column": 132}}}, "severity": "WARNING"}' + +1. Incorporate the suggestions when possible and appropriate. Every docs page contain metadata in the first few lines: From c94deca1190b80cbeec988e9d96ef3406777350a Mon Sep 17 00:00:00 2001 From: Siye Wang Date: Mon, 6 May 2024 13:31:05 +0800 Subject: [PATCH 63/67] update missing Chinese translation for v2.6 --- .../advanced-user-guides.md | 11 + .../manage-project-resource-quotas.md | 50 +++++ .../manage-projects/manage-projects.md | 42 ++++ .../backup-restore-and-disaster-recovery.md | 117 +++++++++++ .../deploy-apps-across-clusters.md | 21 ++ .../helm-charts-in-rancher.md | 188 ++++++++++++++++++ .../horizontal-pod-autoscaler.md | 30 +++ .../kubernetes-resources-setup.md | 73 +++++++ .../load-balancer-and-ingress-controller.md | 65 ++++++ .../workloads-and-pods/workloads-and-pods.md | 82 ++++++++ .../launch-kubernetes-with-rancher.md | 84 ++++++++ .../nutanix/nutanix.md | 21 ++ .../use-new-nodes-in-an-infra-provider.md | 156 +++++++++++++++ .../vsphere/vsphere.md | 60 ++++++ 14 files changed, 1000 insertions(+) create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/advanced-user-guides/advanced-user-guides.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md create mode 100644 i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/advanced-user-guides/advanced-user-guides.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/advanced-user-guides/advanced-user-guides.md new file mode 100644 index 00000000000..80d6b10f321 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/advanced-user-guides/advanced-user-guides.md @@ -0,0 +1,11 @@ +--- +title: 高级用户指南 +--- + + + + + +高级用户指南是“问题导向”的文档，用户可以从中学习如何解决问题。高级用于指南与新用户指南的主要区别在于，高级用户指南面向更有经验或更高级的用户，这些用户对文档有更多的技术需求，而且已经了解 Rancher 及其功能。他们知道自己需要做什么，只是需要额外的指导来完成更复杂的任务。 + +应该注意的是，新用户指南和高级用户指南都没有提供详细的解释或讨论（这些文档不包括在本部分）。操作指南侧重于引导用户通过可重复、有效的步骤来学习新技能、掌握某些操作或解决某些问题。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md new file mode 100644 index 00000000000..ce57b2db349 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/advanced-user-guides/manage-projects/manage-project-resource-quotas/manage-project-resource-quotas.md @@ -0,0 +1,50 @@ +--- +title: 项目资源配额 +--- + + + + + +如果多个团队共享一个集群，某个团队可能会使用过多的可用资源，例如 CPU、内存、存储、服务、Kubernetes 对象（如 Pod 或 Secret）等。你可以应用 _资源配额_ 来防止过度消耗资源。资源配额是 Rancher 用来限制项目或命名空间可用资源的功能。 + +本文介绍如何在现有项目中创建资源配额。 + +你也可以在创建新项目时设置资源配额。有关详细信息，请参阅[创建新项目](../../../new-user-guides/manage-clusters/projects-and-namespaces.md#创建项目)。 + +Rancher 中的资源配额包含与 [Kubernetes 原生版本](https://kubernetes.io/docs/concepts/policy/resource-quotas/)相同的功能。Rancher 还扩展了资源配额的功能，从而让你将资源配额应用于项目。有关资源配额如何与 Rancher 中的项目一起使用的详细信息，请参阅[此页面](about-project-resource-quotas.md)。 + +### 将资源配额应用于现有项目 + +修改资源配额的使用场景如下： + +- 限制某个项目和项目下的命名空间能使用的资源 +- 在资源配额已生效的情况下，对项目可用的资源进行扩容或缩容 + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面，进入要应用资源配额的集群，然后单击 **Explore**。 +1. 单击**集群 > 项目/命名空间**。 +1. 确保 **Projects/Namespaces** 页面处于 **Group by Project** 视图模式。 + ![Screenshot highlighting the "Group by Project" icon, above the list of projects. It resembles a folder.](/img/edit-project-config-for-resource-quotas-group-by-project.png) + +1. 找到要添加资源配额的项目，选择与项目名称同行的 **⋮**。 + ![Screenshot highlighting triple dots icon at the end of the same row as the project name.](/img/edit-project-config-for-resource-quotas-dots.png) + +1. 选择**编辑配置**。 + +1. 展开**资源限额**并单击**添加资源**。你也可以编辑现有配额。 + +1. 选择资源类型。有关类型的更多信息，请参阅[配额类型参考](resource-quota-types.md)。 + +1. 输入**项目限制**和**命名空间默认限制**的值。 + + | 字段 | 描述 | + | ----------------------- | -------------------------------------------------------------------------------------------------------- | + | 项目限制 | 项目的总资源限制。 | + | 命名空间默认限制 | 每个命名空间的默认资源限制。此限制会沿用到项目中的每个命名空间。项目中所有命名空间的限制之和不应超过项目限制。 | + +1. **可选**：添加更多配额。 + +1. 单击**创建**。 + +**结果**：资源配额已应用到你的项目和命名空间。如果你后续需要添加更多命名空间，Rancher 会验证项目是否可以容纳该命名空间。如果项目无法分配资源，你仍然可以创建命名空间，但命名空间将获得的资源配额为 0。然后 Rancher 将不允许你创建任何受此配额限制的资源。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md new file mode 100644 index 00000000000..46637b053a4 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/advanced-user-guides/manage-projects/manage-projects.md @@ -0,0 +1,42 @@ +--- +title: 项目管理 +--- + + + + + +_项目_ 是 Rancher 中引入的对象，可帮助你更有组织地管理 Kubernetes 集群中的命名空间。你可以使用项目创建多租户集群，这种集群允许一组用户共享相同的底层资源来创建应用，而应用之间不会相互影响。 + +在层次结构方面： + +- 集群包含项目 +- 项目包含命名空间 + +在 Rancher 中，你可以使用项目将多个命名空间作为一个实体进行管理。在原生 Kubernetes（没有项目这个概念）中，RBAC 或集群资源等功能被分配给了各个命名空间。如果集群中的多个命名空间需要分配同样的访问权限，分配权限会变得非常繁琐。即使所有命名空间都需要相同的权限，但也无法使用一个操作中将这些权限应用于所有命名空间。你必须重复地将这些权限分配给每个命名空间。 + +而 Rancher 通过引入项目的概念，通过允许你在项目级别应用资源和访问权限。然后，项目中的每个命名空间都会继承这些资源和策略。因此你只需将资源和策略分配给项目即可，不需要将它们分配给每个单独的命名空间。 + +你可以使用项目执行以下操作： + +- [为用户分配一组命名空间的访问权限](../../new-user-guides/add-users-to-projects.md) +- 为用户分配[项目中的特定角色](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色)。角色可以是所有者、成员、只读或[自定义](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/custom-roles.md) +- [设置资源配额](manage-project-resource-quotas/manage-project-resource-quotas.md) +- [管理命名空间](../../new-user-guides/manage-namespaces.md) +- [配置工具](../../../reference-guides/rancher-project-tools.md) +- [持续集成和部署设置流水线](ci-cd-pipelines.md) +- [配置 Pod 安全策略](manage-pod-security-policies.md) + +### 授权 + +非管理者用户只有在[管理员](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/global-permissions.md)、[集群所有者或成员](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#集群角色)或[项目所有者](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色)将非管理员用户添加到项目的**成员**选项卡后，才能获取项目的访问权限。 + +创建项目的人自动成为[项目所有者](../../new-user-guides/authentication-permissions-and-global-configuration/manage-role-based-access-control-rbac/cluster-and-project-roles.md#项目角色)。 + +## 在项目之间切换 + +要在项目之间切换，请使用导航栏中的下拉菜单。你也可以直接在导航栏中切换项目： + +1. 在左上角，单击 **☰ > 集群管理**。 +1. 在**集群**页面，进入要切换项目的集群然后点击 **Explore**。 +1. 在顶部导航栏中，选择要打开的项目。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md new file mode 100644 index 00000000000..a5b8b3ff7ec --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/backup-restore-and-disaster-recovery/backup-restore-and-disaster-recovery.md @@ -0,0 +1,117 @@ +--- +title: 备份和灾难恢复 +keywords: [rancher v2.6 备份还原, rancher v2.6 备份与还原, 备份恢复 rancher v2.6, rancher v2.6 备份与恢复] +--- + + + + + +在本节中，你将学习如何创建 Rancher 的备份，如何从备份中恢复 Rancher，以及如何将 Rancher 迁移到新的 Kubernetes 集群。 + +`rancher-backup` operator 可以用来备份和恢复任何 Kubernetes 集群上的 Rancher。这个应用是一个 Helm Chart，可以通过 Rancher 的 **Apps & Marketplace** (Rancher v2.6.5 之前的版本) 或者 **Apps** (Rancher v2.6.5+) 页面，或使用 Helm CLI 部署。你可以访问[本页面](https://github.com/rancher/charts/tree/release-v2.6/charts/rancher-backup)获取 `rancher-backup` Helm Chart。 + +backup-restore operator 需要安装在 local 集群上，并且只对 Rancher 应用进行备份。备份和恢复操作仅在本地 Kubernetes 集群中执行。 + + +## 备份和恢复 Docker 安装的 Rancher + +对于使用 Docker 安装的 Rancher，请参见[备份](back-up-docker-installed-rancher.md)和[恢复](restore-docker-installed-rancher.md)对 Rancher 进行备份和恢复。 + +## 备份和恢复原理 + +`rancher-backup` operator 引入了三个自定义资源，分别是 Backups、Restores 和 ResourceSets。将以下集群范围的自定义资源定义添加到集群中： + +- `backups.resources.cattle.io` +- `resourcesets.resources.cattle.io` +- `restores.resources.cattle.io` + +ResourceSet 定义了需要备份哪些 Kubernetes 资源。由于备份 Rancher 所需的值是预设的，因此 ResourceSet 无法通过 Rancher UI 进行配置。请不要修改此 ResourceSet。 + +在创建 Backup 自定义资源时，`rancher-backup` operator 调用 `kube-apiserver` 来获取 Backup 自定义资源引用的 ResourceSet（即预设的 `rancher-resource-set`）资源。 + +然后，operator 以 .tar.gz 格式创建备份文件，并将其存储在 Backup 资源中配置的位置。 + +在创建 Restore 自定义资源时，operator 访问 Restore 指定的 tar.gz 备份文件，并从该文件恢复应用。 + +你可以使用 Rancher UI 或 `kubectl apply` 来创建 Backup 和 Restore 自定义资源。 + +:::note + +请参见[此处](migrate-rancher-to-new-cluster.md#2-使用-restore-自定义资源来还原备份)获取在 Rancher 2.6.3 中将现有备份文件恢复到 v1.22 集群的帮助。 + +::: + +## 安装 rancher-backup operator + +你可以使用 Rancher UI 或 Helm CLI 来安装 `rancher-backup` operator。两种安装方法都将 `rancher-backup` Helm Chart 安装在运行 Rancher Server 的 Kubernetes 集群上。它是集群管理员独有的功能，仅适用于 **local** 集群。（*如果你在 Rancher UI 中没有看到 `rancher-backup`，你可能选择了错误的集群。*） + +:::note + +使用 backup-restore-operator 执行恢复后，Fleet 中会出现一个已知问题：用于 clientSecretName 和 helmSecretName 的密文不包含在 Fleet 的 Git 仓库中。请参见[此处](../../../integrations-in-rancher/fleet-gitops-at-scale/fleet-gitops-at-scale.md#故障排除)获得解决方法。 + +::: + +### 使用 Rancher UI 安装 rancher-backup + + + + +1. 在左上角，单击 **☰ > Cluster Management**。 +1. 在 **Clusters** 页面上，转到 `local` 集群并单击 **Explore**。 +1. 在左侧导航栏中，单击 **Apps > Charts**。 +1. 点击 **Rancher 备份**。 +1. 单击**安装**。 +1. 可选：配置默认存储位置。如需获取帮助，请参见[配置](../../../reference-guides/backup-restore-configuration/storage-configuration.md)。 +1. 单击**安装**。 + + + + +1. 在左上角，点击 **☰ > Cluster Management**。 +1. 在 **Clusters** 页面, 转到 `local` 集群并点击 **Explore**。 +1. 在左侧导航栏中，选择, **Apps & Marketplace > Charts**。 +1. 点击 **Rancher Backups**。 +1. 点击 **Install**。 +1. 可选：配置默认存储位置。需要帮助，请参阅 [configuration 章节。](../../../reference-guides/backup-restore-configuration/storage-configuration.md) +1. 点击 **Install**。 + + + + +**结果**：`rancher-backup` operator 已安装。 + +在**集群仪表板**中，你可以看到列在 **Deployments** 下的 `rancher-backup` operator。 + +如果需要在 Rancher 中配置备份应用，在左侧导航栏中单击 **Rancher 备份**。 + +### RBAC + +只有 Rancher 管理员和本地集群的所有者可以： + +* 安装 Chart +* 看到 Backup 和 Restore CRD 的导航链接 +* 通过分别创建 Backup CR 和 Restore CR 执行备份和恢复 +* 列出目前已执行的备份和恢复操作 + +## 备份 Rancher + +备份是通过创建 Backup 自定义资源实现的。如需查看教程，请参见[本页面](back-up-rancher.md)。 + +## 还原 Rancher + +还原是通过创建 Restore 自定义资源实现的。如需查看教程，请参见[本页面](restore-rancher.md)。 + +## 将 Rancher 迁移到新集群 + +你可以按照[这些步骤](migrate-rancher-to-new-cluster.md)执行迁移。 + +## 默认存储位置配置 + +配置一个默认存储位置，所有备份都将保存在其中。您将可以选择在每个备份中覆盖此选项，但仅限于使用与 S3 兼容或 Minio 对象存储。 + +如需了解各个选项的配置，请参见[本页面](../../../reference-guides/backup-restore-configuration/storage-configuration.md)。 + +### rancher-backup Helm Chart 的示例 values.yaml + +当使用 Helm CLI 安装时，可以使用示例 [values.yaml 文件](../../../reference-guides/backup-restore-configuration/storage-configuration.md#rancher-backup-helm-chart-的示例-valuesyaml) 来配置 `rancher-backup` operator 。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md new file mode 100644 index 00000000000..6c4807d291b --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/deploy-apps-across-clusters/deploy-apps-across-clusters.md @@ -0,0 +1,21 @@ +--- +title: 跨集群部署应用 +--- + + + + + +不同版本的 Rancher 提供了几种不同的方式来部署跨集群应用。 + +## Fleet + +Rancher v2.5 及更高版本使用 Fleet 跨集群部署应用 + +使用 Fleet 的持续交付是大规模的 GitOps。如需更多信息，请参阅 [Fleet](fleet.md)。 + +### 多集群应用 + +在 v2.5 之前的 Rancher 中，多集群应用功能用于跨集群部署应用。多集群应用功能已弃用，但仍可作为旧版功能使用。 + +详情请参阅[此文档](multi-cluster-apps.md)。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md new file mode 100644 index 00000000000..48ff5338869 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md @@ -0,0 +1,188 @@ +--- +title: Helm Charts 和 Apps +--- + + + + + +在本节中，你将学习如何在 Rancher 中管理 Helm Chart 仓库和应用。 + +## Helm Charts 在 Rancher 中的工作原理 + +Rancher 使用应用商店系统导入一系列的 charts 包到仓库里，然后使用这些 charts 来部署自定义的 Kubernetes 应用程序或 Rancher 的工具，如监控（Monitoring）或 Istio。Rancher 工具预先集成进了仓库，并通过独立的 Helm chart 进行部署。如果你有自己的额外仓库只需要添加到当前的集群中就可以正常部署。 + +### Catalogs, Apps, Rancher UI + +[在 Rancher v2.4 及更早版本中](/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/helm-charts-in-rancher/helm-charts-in-rancher.md), 存储部署的应用程序的仓库被称为 "catalogs"。这些仓库是通过 UI 的 **Catalogs** 进行管理的。 + +Helm chart 仓库现在使用 **Apps & Marketplace**（Rancher v2.6.5之前）或 **Apps**（Rancher v2.6.5及以后版本）进行管理。 + +### Rancher v2.6 中的版本变更 + +从 Rancher v2.6.0 开始，为 Rancher chart 实施了新的版本设计方案。方案主要围绕 charts 的主要版本以及与上游 charts 相关的 `+up` 注释进行设计。 + +**主要版本:** charts 的主要版本与 Rancher 的次要版本相关联。当你升级到新的 Rancher 次要版本时，你应该确保所有的 charts 也被升级对应的主版本。 + +**基于上游的 Charts:** 在升级时，确保上游 chart 的版本与你的 Rancher 版本兼容。chart 的 `+up` 注释指示 Rancher chart 正在跟踪的上游版本。例如，`100.x.x+up16.6.0` 的监控（Monitoring）跟踪上游的 kube-prometheus-stack `16.6.0` 并附带一些额外的 Rancher 补丁。 + +在升级 Rancher 版本时，不要降级你正在使用的 chart 版本。例如，如果你在 Rancher v2.5 中使用的监控版本比 `16.6.0` 还要高，你不应该升级到 `100.x.x+up16.6.0`，而是应该在下一个发布中升级到适当的版本。 + +:::note + +下表中提及的主要版本之下的任何版本均仅适用于 2.5 及以下版本。例如，在 2.6.x 及以上版本中，建议您不要使用 <100.x.x 版本的监控。 + +::: + +### 版本控制方案 + +| **Name** | **支持的最低版本** | **支持的最高版本** | +| ---------------- | ------------ | ------------ | +| external-ip-webhook | 100.0.0+up1.0.0 | 100.0.1+up1.0.1 | +| harvester-cloud-provider | 100.0.2+up0.1.12 | 100.0.2+up0.1.12 | +| harvester-csi-driver | 100.0.2+up0.1.11 | 100.0.2+up0.1.11 | +| neuvector | 100.0.0+up2.2.0 | 100.0.0+up2.2.0 | +| rancher-alerting-drivers | 100.0.0 | 100.0.2 | +| rancher-backup | 2.0.1 | 2.1.2 | +| rancher-cis-benchmark | 2.0.1 | 2.0.4 | +| rancher-gatekeeper | 100.0.0+up3.6.0 | 100.1.0+up3.7.1 | +| rancher-istio | 100.0.0+up1.10.4 | 100.3.0+up1.13.3 | +| rancher-logging | 100.0.0+up3.12.0 | 100.1.2+up3.17.4 | +| rancher-longhorn | 100.0.0+up1.1.2 | 100.1.2+up1.2.4 | +| rancher-monitoring | 100.0.0+up16.6.0 | 100.1.2+up19.0.3 | +| rancher-sriov (experimental) | 100.0.0+up0.1.0 | 100.0.3+up0.1.0 | +| rancher-vsphere-cpi | 100.3.0+up1.2.1 | 100.3.0+up1.2.1 | +| rancher-vsphere-csi | 100.3.0+up2.5.1-rancher1 | 100.3.0+up2.5.1-rancher1 | +| rancher-wins-upgrader | 0.0.100 | 100.0.1+up0.0.1 | + +## 访问 Charts + + + + +从左上角菜单中选择 **Apps** 然后您将进入 **Charts** 页面。 + + + + +从左上角菜单中选择 **Apps & Marketplace** 然后您将进入 **Charts** 页面。 + + + + +**Charts** 页面包含所有 Rancher、Rancher 合作伙伴和自定义 Chart。你可以通过选择左侧的下拉菜单来筛选 Chart： + +* Rancher 工具（例如 Logging 或 Monitoring）包含在 **Rancher** 标签下 +* Partner Chart 位于 **Partner** 标签下 +* 自定义 Chart 将显示在仓库的名称下 + +所有这三种类型都以相同的方式部署和管理。 + +:::note + +在传统的 Rancher UI 中由集群管理器管理的应用程序（全局视图）应继续只由集群管理器管理，在新 UI 中使用 **Apps & Marketplace** （Rancher v2.6.5 之前）或 **Apps**（Rancher v2.6.5+）管理的应用程序必须仅由 **Apps & Marketplace** 或 **Apps** 管理。 + +::: + +### 管理仓库 + +**Repositories** 页面列出了你的 Helm 仓库。 这包括传统的具有 index.yaml 的 Helm 端点，以及克隆并指向特定分支的 Git 仓库。要使用自定义 Charts，在这里可以添加你的仓库。添加仓库后，你可以在 **Charts** 页面中访问自定义 Charts，这些 Charts 将列在仓库的名称下。 + +访问 **Repositories**，从左侧导航菜单中选择 **Repositories** 。 + +### 添加自定义 Git 仓库 + +要添加一个包含你的 Helm Charts 或集群模板定义的自定义 Git 仓库： + +1. 从左侧导航菜单中选择 **Repositories**。 +2. 点击 **Create**。 +3. 选择 **Git repository containing Helm chart...**。 +4. 您必须输入一个名称和一个 Git仓 库 URL。 其他字段，包括描述，都是可选的。如果您不想从仓库所有者设置为默认的分支中拉取，请输入一个备用分支名称。通常，默认分支命名为 `main` 或 `master`。 +5. 点击 **Create** 添加仓库。 + +在 Rancher 中添加 Charts 仓库后，它将立即生效。 + +### 添加自定义 Helm Chart 仓库 + +你可以将自己的 Helm chart 仓库添加到 Rancher。为了能正确添加 http 的 Helm Chart 仓库，你需要提供 chart 的服务器并能够响应 GET 请求并提供 YAML 文件和 tar 包。 + +有关 Helm chart 仓库的更多信息，请参阅 [官方 Helm 文档](https://helm.sh/docs/topics/chart_repository/)。 + +1. 从左侧导航菜单中选择 **Repositories** 。 +1. 点击 **Create**。 +1. 选择 **http(s) URL to an index generated by Helm**。 +1. 输入一个仓库名称和 chart repository 的 index URL 地址。 + +### 添加私有 Git/Helm Chart 仓库 + +你可以使用 SSH 密钥凭据或 HTTP 基础认证秘密（如用户名和密码）添加私有 Git 或 Helm chart 仓库。 + +### 向仓库添加私有 CA + +将私有 CA 添加到 Helm chart 仓库： + +1. 从左侧导航菜单中选择 **Repositories**。 +2. 找到你想要向其添加私有 CA 证书的 Git 或 HTTP 的仓库。点击 **⋮ > Edit YAML**。 +3. 设置 `caBundle` 值，如以下示例： + + ```yaml + [...] + spec: + caBundle: + MIIFXzCCA0egAwIBAgIUWNy8WrvSkgNzV0zdWRP79j9cVcEwDQYJKoZIhvcNAQELBQAwPzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMRQwEgYDVQQKDAtNeU9yZywgSW5jLjENMAsGA1UEAwwEcm9vdDAeFw0yMTEyMTQwODMyMTdaFw0yNDEwMDMwODMyMT + ... + nDxZ/tNXt/WPJr/PgEB3hQdInDWYMg7vGO0Oz00G5kWg0sJ0ZTSoA10ZwdjIdGEeKlj1NlPyAqpQ+uDnmx6DW+zqfYtLnc/g6GuLLVPamraqN+gyU8CHwAWPNjZonFN9Vpg0PIk1I2zuOc4EHifoTAXSpnjfzfyAxCaZsnTptimlPFJJqAMj+FfDArGmr4= + [...] + ``` + +:::note 带有认证的 Helm chart 仓库 + +从 Rancher v2.6.3 开始，Repo.Spec 中添加了一个新值 `disableSameOriginCheck` ，这允许用户绕过同源检查，在所有 API 调用中以 Basic Auth Header 的形式发送仓库身份验证信息。这并不推荐，但在非标准 Helm chart 仓库的情况下，例如具有重定向到不同源 URL 的仓库，可以作为临时解决方案使用。 + +要将此功能用于现有 Helm Chart，点击 **⋮ > Edit YAML**。在 YAML 文件的 `spec` 部分，添加 `disableSameOriginCheck` 并将其设置为 `true`： + +```yaml +[...] +spec: + disableSameOriginCheck: true +[...] +``` + +::: + +### Helm 兼容性 + +仅支持 Helm 3 兼容 Chart 。 + +### 部署和升级 Chart + +从 **Charts** 标签页中选择要安装的 chart。Rancher 和合作伙伴 chart 可能通过自定义页面或 questions.yaml 文件提供额外的配置，但所有chart 安装都可以修改 values.yaml 和其他基本设置。点击安装后，将部署一个 Helm 操作作业，并显示作业的控制台。 + +要查看所有最近的更改，请转到 **Recent Operations** 标签页。从那里，您可以查看所进行的调用、条件、事件和日志。 + +安装 chart 后，您可以在 **Installed Apps** 标签页中找到它。在本节中，你可以升级或删除安装，并查看更多详细信息。选择升级时，呈现的形式和数值与安装相同。 + +大多数 Rancher 工具在 **Apps & Marketplace** 下方的工具栏中都有额外的页面，以帮助你管理和使用这些功能。这些页面包括指向仪表板的链接、可轻松添加自定义资源的表单以及其他信息。 + +:::caution + +如果在升级前使 **自定义 Helm 选项**，并且你的 Chart 中包含不可更改的字段，请注意使用 `--force` 选项可能会导致错误，如果您的 Chart 具有不可变字段。这是因为 Kubernetes 中的某些对象一旦创建就无法更改。要避免该错误，你可以： + +* 使用默认升级选项（即不要使用 `--force` 选项） +* 卸载现有 Chart 并安装升级后的 Chart +* 在执行 `--force` 升级之前，从集群中删除具有不可变字段的资源。 + +::: + +#### Rancher v2.6.3 的变更 + +在 **Apps > Installed Apps** 页面上，旧版应用没有升级按钮。 + +如果你想升级已安装的旧版应用， 必须启用[旧版功能](../../advanced-user-guides/enable-experimental-features/enable-experimental-features.md)。 如果你在升级 Rancher 之前已经运行了旧版应用，此标志会自动启用。 + +- 您可以从集群资源管理器中，从左侧导航栏的 **Legacy > Project > Apps** 部分升级该应用。 +- 对于多集群应用程序，您可以跳转到 **≡ > Multi-cluster Apps** 并从那里升级应用程序。 + +### 限制 + +Rancher CLI 不能用于安装仪表板应用程序或 Rancher 功能 Chart。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md new file mode 100644 index 00000000000..b50aecb39a9 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/horizontal-pod-autoscaler/horizontal-pod-autoscaler.md @@ -0,0 +1,30 @@ +--- +title: Horizontal Pod Autoscaler +description: 了解 Pod 水平自动扩缩 (HPA)。如何管理 HPA 以及如何使用服务部署来进行测试 +--- + + + + + +[Horizontal Pod Autoscaler（HPA）](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)是一项 Kubernetes 功能，用于将集群配置为自动扩缩其运行的服务。 + +Rancher 提供了一些附加功能来帮助管理 HPA，具体取决于 Rancher 的版本。 + +你可以使用 Rancher UI 创建、管理和删除 HPA。仅在 `autoscaling/v2beta2` API 中支持 HPA。 + +## 管理 HPA + +管理 HPA 的方式因你的 Kubernetes API 版本而异： + +- **Kubernetes API 版本 autoscaling/V2beta1**：允许根据应用程序的 CPU 和内存利用率自动扩缩 pod。 +- **Kubernetes API 版本 autoscaling/V2beta2**：允许根据 CPU 和内存利用率以及自定义指标自动扩缩 pod。 + +你可以使用 Rancher UI 创建、管理和删除 HPA。在 Rancher UI 中，你可以将 HPA 配置为根据 CPU 和内存利用率进行扩缩。有关详细信息，请参阅[使用 Rancher UI 管理 HPA](manage-hpas-with-ui.md)。如需根据自定义指标进行 HPA，你仍然需要使用 `kubectl`。有关详细信息，请参阅[配置 HPA 以使用 Prometheus 自定义指标进行扩缩](manage-hpas-with-kubectl.md#使用资源指标cpu-和内存配置-hpa-以进行扩缩)。 + +在 Rancher 2.0.7 及更高版本中创建的集群自动满足使用 HPA 的所有要求（metrics-server 和 Kubernetes 集群配置）。 +## 使用服务部署测试 HPA + +你可以转到你的项目并单击 **资源 > HPA** 来查看​​ HPA 当前的副本数。有关详细信息，请参阅[获取 HPA 指标和状态](manage-hpas-with-ui.md)。 + +你还可以使用 `kubectl` 来获取你使用负载测试工具测试的 HPA 的状态。有关详细信息，请参阅[使用 kubectl 测试 HPA](test-hpas-with-kubectl.md)。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md new file mode 100644 index 00000000000..a286afadd9f --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/kubernetes-resources-setup.md @@ -0,0 +1,73 @@ +--- +title: Kubernetes 资源 +--- + + + + + +你可以在 Rancher UI 中查看和操作 Kubernetes 集群中的所有自定义资源和 CRD。 + +## 工作负载 + +使用[工作负载](workloads-and-pods/workloads-and-pods.md)将应用部署到集群节点，工作负载是包含用于运行应用的 pod 的对象，以及为部署行为设置规则的元数据。工作负载可以部署在集群范围内，也可以部署在一个命名空间内。 + +部署工作负载时，你可以使用任何镜像进行部署。可供选择的[工作负载类型](workloads-and-pods/workloads-and-pods.md#工作负载类型)有多种，工作负载类型决定了你的应用程序的运行方式。 + +在工作负载部署之后，你可以继续使用它。你可以： + +- 将工作负载[升级](workloads-and-pods/upgrade-workloads.md)到它运行的应用的更新版本。 +- 如果升级出现问题，将工作负载[回滚](workloads-and-pods/roll-back-workloads.md)到以前的版本。 +- [添加一个 sidecar](workloads-and-pods/add-a-sidecar.md)，这是一个支持主要工作负载的工作负载。 + +## 负载均衡和 Ingress + +### 负载均衡器 + +启动应用程序后，它仅在集群中可用。无法从外部访问它。 + +如果你希望你的应用程序可以从外部访问，则必须向集群添加负载均衡器。如果用户知道负载均衡器的 IP 地址和应用的端口号，负载均衡器可以为外部连接创建一个访问集群的网关。 + +Rancher 支持两种类型的负载均衡器： + +- [Layer-4 负载均衡器](load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing.md#layer-4-load-balancer#四层负载均衡器) +- [Layer-7 负载均衡器](load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing.md#七层负载均衡器) + +有关详细信息，请参阅[负载均衡器](load-balancer-and-ingress-controller/layer-4-and-layer-7-load-balancing.md)。 + +#### Ingress + +负载均衡器只能处理每个 service 的一个 IP 地址。换言之，如果你在集群中运行了多个 service，则必须为每个 service 配备一个负载均衡器。运行多个负载均衡器的花费可能非常高昂。因此，你可以使用 Ingress 来解决此问题。 + +Ingress 是一组充当负载均衡器的规则。Ingress 与一个或多个 Ingress Controller 一起动态路由 service 的请求。Ingress 收到请求时，集群中的 Ingress Controller 会对负载均衡器进行配置，从而根据你配置的 service 子域或路径规则将请求定向到正确的 service。 + +有关详细信息，请参阅 [Ingress](load-balancer-and-ingress-controller/add-ingresses.md)。 + +在项目中使用 Ingress 时，你可以通过设置全局 DNS 条目来将 Ingress 主机名编程到外部 DNS。 + +## 服务发现 + +使用负载均衡器和/或 Ingress 将集群公开给外部请求后，你只能通过 IP 地址访问集群。要创建可解析的主机名，你必须创建服务记录，该记录将 IP 地址、外部主机名、DNS 记录别名、工作负载或标记的 pod 映射到特定主机名。 + +有关详细信息，请参阅[服务发现](create-services.md)。 + +## 流水线 + +在将您的项目[配置到版本控制提供商](../../../reference-guides/pipelines/pipelines.md#1-配置版本控制提供程序)之后，您可以添加仓库并开始为每个仓库配置流水线。 + +更多信息，请[参阅](../../../reference-guides/pipelines/pipelines.md)。 + +## 应用程序 + +除了启动应用程序的各个组件外，你还可以使用 Rancher 应用商店来启动应用，即 Helm Chart。 + +## Kubernetes 资源 + +在 Rancher 项目或命名空间的上下文中，_资源_ 是支持 Pod 操作的文件和数据。在 Rancher 中，证书、镜像仓库和密文都被视为资源。但是，Kubernetes 将资源划分为不同类型的[密文（secret）](https://kubernetes.io/docs/concepts/configuration/secret/)。因此，在单个项目或命名空间中，各个资源必须具有唯一的名称以避免冲突。资源主要用于承载敏感信息，但也有其他用途。 + +资源包括： + +- [证书](encrypt-http-communication.md)：用于加密/解密进入或离开集群的数据的文件。 +- [ConfigMap](configmaps.md)：存储一般配置信息的文件，例如一组配置文件。 +- [密文](secrets.md)：存储密码、token 或密钥等敏感数据的文件。 +- [镜像仓库](kubernetes-and-docker-registries.md)：携带用于验证私有镜像仓库的凭证的文件。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md new file mode 100644 index 00000000000..00ae5a04894 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/load-balancer-and-ingress-controller/load-balancer-and-ingress-controller.md @@ -0,0 +1,65 @@ +--- +title: 在 Rancher 中设置负载均衡器和 Ingress Controller +description: 了解如何设置负载均衡器和 Ingress Controller 以在 Rancher 中重定向服务请求，并了解负载均衡器的限制 +--- + + + + + +在 Rancher 中，你可以通过设置负载均衡器和 Ingress Controller 来重定向服务请求。 + +## 负载均衡器 + +启动应用程序后，该应用程序仅在集群内可用。你无法从集群外部访问它。 + +如果你希望从外部访问应用程序，则必须向集群添加负载均衡器或 Ingress。如果用户知道负载均衡器的 IP 地址和应用的端口号，负载均衡器可以为外部连接创建一个访问集群的网关。 + +Rancher 支持两种类型的负载均衡器： + +- [Layer-4 负载均衡器](layer-4-and-layer-7-load-balancing.md#四层负载均衡器) +- [Layer-7 负载均衡器](layer-4-and-layer-7-load-balancing.md#七层负载均衡器) + +有关详细信息，请参阅[负载均衡器](layer-4-and-layer-7-load-balancing.md)。 + +### 负载均衡器限制 + +负载均衡器有几个需要注意的限制： + +- 负载均衡器只能处理每个 service 的一个 IP 地址。换言之，如果你在集群中运行了多个 service，则必须为每个 service 配备一个负载均衡器。运行多个负载均衡器的花费可能非常高昂。 + +- 如果你想将负载均衡器与托管的 Kubernetes 集群（即托管在 GKE、EKS 或 AKS 中的集群）一起使用，则负载均衡器必须运行在该云提供商的基础设施上。请根据你配置集群的方式查看负载均衡器的兼容列表： + +- [支持 Layer-4 负载均衡](layer-4-and-layer-7-load-balancing.md#四层负载均衡支持) + +- [支持 Layer-7 负载均衡](layer-4-and-layer-7-load-balancing.md#七层负载均衡支持) + +## Ingress + +如上所述，使用负载均衡器的缺点是： + +- 每个服务负载均衡器只能处理一个 IP 地址。 +- 如果你在集群中运行多个服务，则每个服务都必须配备一个负载均衡器。 +- 为每个服务配备负载均衡器的花费可能非常高昂。 + +相反，如果将某个 Ingress 用作集群的入口点，Ingress 可以更灵活地将流量路由到多个 service。它可以将多个 HTTP 请求映射到 service，而无需为每个 service 提供单独的 IP 地址。 + +因此，如果你需要使用相同的 IP 地址、Layer 7 协议或特权节点端口（80 和 443）来公开多个 service，你可以使用一个 Ingress。 + +Ingress 与一个或多个 Ingress Controller 一起动态路由 service 的请求。Ingress 收到请求时，集群中的 Ingress Controller 会根据你配置的 service 子域或路径规则将请求定向到正确的 service。 + +每个 Kubernetes Ingress 资源都对应一个 `/etc/nginx/sites-available/` 中的文件，其中包含一个配置对特定文件和文件夹的请求的 `server{}` 配置块。 + +Ingress 能为你的集群创建一个入口端口（与负载均衡器类似），可以位于集群的内部或外部。RKE 启动的集群中的 Ingress 和 Ingress Controller 由 [Nginx](https://www.nginx.com/) 提供支持。 + +Ingress 还支持其他功能，例如 SSL 终止、基于名称的虚拟主机等。 + +:::note 在高可用性配置中使用 Rancher： + +请避免将 Ingress 添加到 `local` 集群。Rancher 将 Nginx Ingress Controller 作为 Rancher 管理的 _所有_ 集群的全局入口点，其中包括 `local` 集群。因此，当用户尝试访问应用程序时，Rancher 可能会由于重新加载 Nginx 配置而断开连接。要解决这个问题，我们建议你仅在通过 Rancher 启动的集群中部署应用程序。 + +::: + +- 有关如何在 Rancher 中设置 Ingress 的更多信息，请参阅 [Ingress](add-ingresses.md)。 +- 有关 Ingress 和 Ingress Controller 的完整信息，请参阅 [Kubernetes Ingress 文档](https://kubernetes.io/docs/concepts/services-networking/ingress/)。 +- 在项目中使用 Ingress 时，你可以通过设置全局 DNS 条目来将 Ingress 主机名编程到外部 DNS。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md new file mode 100644 index 00000000000..08be0859bb8 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/kubernetes-resources-setup/workloads-and-pods/workloads-and-pods.md @@ -0,0 +1,82 @@ +--- +title: "Kubernetes 工作负载和 Pod" +description: "了解在 Kubernetes 中构建复杂容器化应用程序的两种结构：Kubernetes 工作负载和 Pod" +--- + + + + + +你可以使用两种基本结构（pod 和工作负载）在 Kubernetes 中构建复杂的容器化应用程序。构建应用程序后，你可以使用第三种结构（service）在集群中或互联网上公开应用程序。 + +### Pod + +[_Pod_](https://kubernetes.io/docs/concepts/workloads/pods/pod-overview/) 是一个或多个共享网络命名空间和存储卷的容器。大多数 pod 只有一个容器。因此，我们讨论的 _pod_ 通常等同于 _容器_。扩展 pod 的方式与扩展容器的方式相同，即配置实现服务的同一 pod 的多个实例。通常，Pod 会根据工作负载进行扩展和管理。 + +### 工作负载 + +_工作负载_ 是为 pod 设置部署规则的对象。Kubernetes 基于这些规则执行部署，并根据应用程序的当前状态来更新工作负载。 +工作负载让你可以定义应用程序调度、扩展和升级的规则。 + +#### 工作负载类型 + +Kubernetes 将工作负载分为不同的类型。Kubernetes 支持的最流行的类型是： + +- [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) + + _Deployment_ 最适合用于无状态应用程序（即不需要维护工作负载的状态）。由 Deployment 类型工作负载管理的 Pod 是独立且一次性的。如果 pod 中断了，Kubernetes 会删除该 pod 然后重新创建它。一个示例应用程序是 Nginx Web 服务器。 + +- [StatefulSet](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) + + 与 Deployment 相比，_StatefulSet_ 最适合在需要维护身份和存储数据的应用程序中使用。适用的应用程序类似于 Zookeeper（一个需要数据库进行存储的应用程序）。 + +- [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) + + _Daemonset_ 确保集群中的每个节点都运行 pod 的副本。如果你需要收集日志或监控节点性能，这种类似 daemon 的工作负载效果是最好的。 + +- [Job](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) + + _Job_ 启动一个或多个 Pod 并确保指定数量的 Pod 能成功终止。Job 最好用于运行有限任务至完成状态，而不是管理正在进行的应用程序的所需状态。 + +- [CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) + + _CronJobs_ 与 Job 类似。但是，CronJob 会基于 cron 的计划运行到完成状态。 + +### Services + +在许多用例中，工作负载必须： + +- 由集群中的其他工作负载访问。 +- 暴露给外部。 + +你可以通过创建一个 _Service_ 实现这些目的。Service 使用[选择器/标签（查看代码示例）](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#service-and-replicationcontroller)来映射到底层工作负载的 pod。Rancher UI 使用你选择的服务端口和类型来自动创建 service 以及工作负载，从而简化此映射过程。 + +#### Service 类型 + +Rancher 中有几种可用的 Service 类型。以下描述来自 [Kubernetes 文档](https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types)。 + +- **ClusterIP** + + > 在集群内部 IP 上公开 Service。如果你选择此值，Service 只能从集群内访问。这是默认的 `ServiceType`。 + +- **NodePort** + + > 在每个节点 IP 上的静态端口（`NodePort`）上暴露 Service。`ClusterIP` service 是自动创建的，而 `NodePort` service 会路由到 ClusterIP service。你可以通过请求 `:` 在集群外部联系 `NodePort` service。 + +- **LoadBalancer** + + > 使用云提供商的负载均衡器向外部公开服务。`NodePort` 和 `ClusterIP` service 是自动创建的，外部负载均衡器会路由到这些 service。 + +## 工作负载选项 + +以下文档介绍了如何部署工作负载和使用工作负载选项。 + +- [部署工作负载](deploy-workloads.md) +- [升级工作负载](upgrade-workloads.md) +- [回滚工作负载](roll-back-workloads.md) + +## 相关链接 + +### 外部链接 + +- [Service](https://kubernetes.io/docs/concepts/services-networking/service/) diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md new file mode 100644 index 00000000000..cc4e205c941 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/launch-kubernetes-with-rancher.md @@ -0,0 +1,84 @@ +--- +title: 使用 Rancher 启动 Kubernetes +--- + + + + + +Rancher 可以使用任意节点启动 Kubernetes 集群。在 Rancher 中将 Kubernetes 部署到这些节点上时，你可以选择 [Rancher Kubernetes Engine](https://rancher.com/docs/rke/latest/en/) (RKE) 或 [RKE2](https://docs.rke2.io) 发行版。Rancher 可以在任何计算机上启动 Kubernetes，包括： + +- 裸金属服务器 +- 本地虚拟机 +- 由云厂商托管的虚拟机 + +Rancher 可以在现有节点上安装 Kubernetes，也可以在云厂商中动态配置节点并安装 Kubernetes。 + +Rancher 还可以创建节点池。在托管在云厂商的节点池上安装 Kubernetes 的一个好处是，如果一个节点与集群断开连接，Rancher 可以自动创建另一个节点并将其加入集群，从而确保节点池的数量符合要求。 + +## RKE + +### 要求 + +如果你使用 RKE 建立集群，节点必须满足下游集群的[节点要求](../kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md)。 + +### 在云厂商的新节点上启动 Kubernetes + +使用 Rancher，你可以基于[节点模板](use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md#节点模板)创建节点池。此节点模板定义了要用于在云厂商中启动节点的参数。 + +在托管在云厂商的节点池上安装 Kubernetes 的一个好处是，如果一个节点与集群断开连接，Rancher 可以自动创建另一个节点并将其加入集群，从而确保节点池的数量符合要求。 + +有关详细信息，请参阅[在新节点上启动 Kubernetes](use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md)。 + +### 在现有自定义节点上启动 Kubernetes + +在这种情况下，你希望将 Kubernetes 安装到裸机服务器、本地虚拟机或云厂商中已存在的虚拟机上。使用此选项，你将在主机上运行 Rancher Agent Docker 容器。 + +如果要重复使用之前的自定义集群中的节点，请在复用之前[清理节点](../manage-clusters/clean-cluster-nodes.md)。如果你重复使用尚未清理的节点，则集群配置可能会失败。 + +有关详细信息，请参阅[自定义节点](../../../reference-guides/cluster-configuration/rancher-server-configuration/use-existing-nodes/use-existing-nodes.md)。 + +### 以编程方式创建 RKE 集群 + +通过 Rancher 以编程方式部署 RKE 集群的最常见方法是使用 Rancher 2 Terraform Provider。详情请参见[使用 Terraform 创建集群](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster)。 + +## RKE2 + +Rancher 2.6 支持直接使用 Rancher UI 配置 [RKE2](https://docs.rke2.io/) 集群。RKE2，也称为 RKE Government，是一个完全符合标准的 Kubernetes 发行版，它专注于安全性和合规性。在 Rancher 2.6.4 及更早版本中，RKE2 配置还处于技术预览阶段。 + +在 Rancher 2.6.5 中，RKE2 已经 GA。 + +### 要求 + +如果你使用 RKE2 建立集群，节点必须满足下游集群的[节点要求](https://docs.rke2.io/install/requirements)。 + +### 在云厂商的新节点上启动 Kubernetes + +RKE2 基于使用上游[集群 API](https://github.com/kubernetes-sigs/cluster-api) 项目的新配置框架。这个新配置框架支持： + +- 将 RKE2 集群配置到 Rancher 具有主机驱动的任何提供商上 +- 完全在 Rancher 中配置 RKE2 集群 +- 除了 Canal 之外，还可以选择 CNI 选项， Calico、Cilium 和 Multus + +RKE2 配置还包括在具有 Windows 节点的集群上安装 RKE2。 + +RKE2 的 Windows 功能包括： + +- Windows 支持 vSphere 主机驱动 +- 用于 Windows RKE2 自定义集群的 Calico CNI +- Calico 的项目网络隔离 (PNI) +- 由 containerd 提供支持的使用 RKE2 的 Windows 容器 +- 通过 Terraform 配置 Windows RKE2 集群 +- 直接从 Rancher UI 配置 Windows RKE2 自定义集群 + +要使 Windows 支持 RKE2 自定义集群，请选择 Calico 作为 CNI。 + +### 在现有自定义节点上启动 Kubernetes + +RKE2 还支持在预配置的虚拟机或裸机节点上安装自定义集群。 + +如果要重复使用之前的自定义集群中的节点，请在复用之前清理节点。如果你重复使用尚未清理的节点，则集群配置可能会失败。 + +### 以编程方式创建 RKE2 集群 + +通过 Rancher 以编程方式部署 RKE2 集群的最常见方法是使用 Rancher 2 Terraform Provider。详情请参见[使用 Terraform 创建集群](https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/cluster_v2)。 \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md new file mode 100644 index 00000000000..885bbb38847 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/nutanix/nutanix.md @@ -0,0 +1,21 @@ +--- +title: 创建 Nutanix AOS 集群 +description: 使用 Rancher 创建 Nutanix AOS (AHV) 集群。集群可能包括具有不同属性的 VM 组，这些属性可用于细粒度控制节点的大小。 +--- + + + + + +[Nutanix Acropolis 操作系统](https://www.nutanix.com/products/acropolis) (Nutanix AOS) 是适用于 Nutanix 超融合基础架构平台的操作系统。AOS 带有一个名为 [Acropolis Hypervisor（AHV）](https://www.nutanix.com/products/ahv)的内置虚拟机监控程序。你可以结合使用 Rancher 与 Nutanix AOS (AHV)，从而在本地体验云环境的操作。 + +Rancher 可以在 AOS (AHV) 中配置节点并在其上安装 Kubernetes。在 AOS 中创建 Kubernetes 集群时，Rancher 首先与 Prism Central API 通信来配置指定数量的虚拟机。然后在虚拟机上安装 Kubernetes。 + +Nutanix 集群可能由多组具有不同属性（例如内存或 vCPU 数量）的 VM 组成。这种分组允许对每个 Kubernetes 角色的节点大小进行细粒度控制。 + +- [创建 Nutanix 集群](provision-kubernetes-clusters-in-aos.md#creating-a-nutanix-aos-cluster) +- [配置存储](provision-kubernetes-clusters-in-aos.md) + +## 创建 Nutanix 集群 + +在[本节](provision-kubernetes-clusters-in-aos.md)中，你将学习如何使用 Rancher 在 Nutanix AOS 中安装 [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes 集群。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md new file mode 100644 index 00000000000..2b7a7cbfcae --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/use-new-nodes-in-an-infra-provider.md @@ -0,0 +1,156 @@ +--- +title: 在云厂商的新节点上启动 Kubernetes +--- + + + + + +在 Rancher 中使用节点模板来创建 RKE 或 RKE2 集群时，每个生成的节点池都会显示在新的**主机池**选项卡中。你可以通过执行以下操作来查看主机池： + +1. 点击**☰ > 集群管理**。 +1. 单击 RKE 或 RKE2 集群的名称。 + +## RKE 集群 + +使用 Rancher，你可以基于[节点模板](use-new-nodes-in-an-infra-provider.md#节点模板)创建节点池。此节点模板定义了要用于在基础设施提供商或云厂商中启动节点的参数。 + +在托管在云厂商的节点池上安装 Kubernetes 的一个好处是，如果一个节点与集群断开连接，Rancher 可以自动创建另一个节点并将其加入集群，从而确保节点池的数量符合要求。 + +可用于创建节点模板的云提供商是由[主机驱动](use-new-nodes-in-an-infra-provider.md#主机驱动)决定的。 + +### 节点模板 + +节点模板保存了用于在特定云提供商中配置节点时要使用的参数。这些节点可以从 UI 启动。Rancher 使用 [Docker Machine](https://docs.docker.com/machine/) 来配置这些节点。可用于创建节点模板的云提供商取决于 Rancher 中状态是 Active 的主机驱动。 + +在 Rancher 中创建节点模板后，模板会被保存，以便你可以再次使用该模板来创建节点池。节点模板绑定到你的登录名。添加模板后，你可以将其从用户配置文件中删除。 + +#### 节点标签 + +你可以为每个节点模板添加[标签](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/)，这样，使用节点模板创建的节点都会自动带有这些标签。 + +无效标签会阻止升级，或阻止 Rancher 启动。有关标签语法的详细信息，请参阅 [Kubernetes 文档](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#syntax-and-character-set)。 + +#### 节点污点 + +你可以为每个节点模板添加[污点](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)，这样，使用节点模板创建的节点都会自动带有这些污点。 + +由于污点可以同时添加到节点模板和节点池中，因此如果添加了相同键的污点效果没有冲突，则所有污点都将添加到节点中。如果存在具有相同键但不同效果的污点，则节点池中的污点将覆盖节点模板中的污点。 + +#### 节点模板的管理员控制 + +管理员可以控制所有节点模板。现在，管理员可以维护 Rancher 中的所有节点模板。当节点模板所有者不再使用 Rancher 时，他们创建的节点模板可以由管理员管理，以便继续更新和维护集群。 + +要访问所有节点模板，管理员需要执行以下操作： + +1. 点击 **☰ > 集群管理**。 +1. 单击 **RKE1 配置 > 节点模板**。 + +**结果**：列出所有节点模板。你可以通过单击 **⋮** 来编辑或克隆模板。 + +### 节点池 + +使用 Rancher，你可以基于[节点模板](#节点模板)创建节点池。 + +节点模板定义了节点的配置，例如要使用的操作系统、CPU 数量和内存量。 + +使用节点池的好处是，如果一个节点被销毁或删除，你可以增加 Active 节点的数量来补偿丢失的节点。节点池可以帮助你确保节点池的计数符合要求。 + +每个节点池必须分配一个或多个节点角色。 + +每个节点角色（即 etcd、controlplane 和 worker）都应分配给不同的节点池。虽然你可以将多个节点角色分配给同一个节点池，但不要在生产集群中执行此操作。 + +推荐的设置： + +- 具有 etcd 角色且计数为 3 的节点池 +- 具有 controlplane 角色且计数至少为 2 的节点池 +- 具有 worker 角色且计数至少为 2 的节点池 + +**离线环境中的 RKE1 下游集群节点**： + +默认情况下，在配置 RKE1 下游集群节点时（例如在 vSphere 中），Rancher 会尝试运行 Docker 安装脚本。但是，Rancher Docker 安装脚本在离线环境中会运行失败。要解决此问题，如果 Docker 已预安装到 VM 镜像上，你可以选择在创建节点模板时跳过安装 Docker。为此，你可以在 Rancher UI **引擎选项**下的 `Docker 安装 URL` 下拉列表中选择 **无**。 + +
**引擎选项下拉列表**
+ +![引擎选项下拉列表](/img/node-template-engine-options-rke1.png) + +#### 节点池污点 + +如果你没有在节点模板上定义[污点](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/)，在节点池上添加污点的好处比在节点模板上添加污点更有利，因为您可以在不担心污点是否存在于节点模板上的情况下更换节点模板。 + +每个污点都将自动添加到节点池中已创建的节点。因此，如果你在已有节点的节点池中添加污点，污点不会应用到已有的节点，但是添加到该节点池中的新节点都将获得该污点。 + +如果污点同时添加到节点模板和节点池中，且添加了相同键的污点效果没有冲突，则所有污点都将添加到节点中。如果存在具有相同键但不同效果的污点，则节点池中的污点将覆盖节点模板中的污点。 + +#### 节点自动替换 + +Rancher 可以自动替换节点池中无法访问的节点。如果节点在指定的时间中处于 Inactive 状态，Rancher 将使用该节点池的节点模板来重新创建节点。 + +:::caution + +自我修复节点池的功能帮助你替换无状态应用的 worker 节点。不建议在 master 节点或连接了持久卷的节点的节点池上启用节点自动替换，因为虚拟机会被临时处理。节点池中的节点与集群断开连接时，其持久卷将被破坏，从而导致有状态应用的数据丢失。 + +::: + +节点自动替换基于 Kubernetes 节点控制器工作。节点控制器定期检查所有节点的状态（可通过 `kube-controller` 的 `--node-monitor-period` 标志配置）。一个节点不可访问时，节点控制器将污染该节点。发生这种情况时，Rancher 将开始其删除倒计时。你可以配置 Rancher 等待删除节点的时间。如果在删除倒计时结束前污点没有被删除，Rancher 将继续删除该节点。Rancher 会根据节点池设置的数量来创建新的节点。 + +#### 启用节点自动替换 + +创建节点池时，你可以指定 Rancher 替换无响应节点的等待时间（以分钟为单位）。 + +1. 在创建或编辑集群的表单中，转到**节点池**。 +1. 转到要启用节点自动替换的节点池。在 **Recreate Unreachable After** 字段中，输入 Rancher 在替换节点之前应该等待节点响应的分钟数。 +1. 填写表单的其余部分以创建或编辑集群。 + +**结果** ：已为节点池启用节点自动替换。 + +#### 禁用节点自动替换 + +你可以执行以下步骤从 Rancher UI 禁用节点自动替换： + +1. 点击 **☰ > 集群管理**。 +1. 在**集群**页面上，转到要禁用节点自动替换的集群，然后单击 **⋮ > 编辑配置**。 +1. 在**节点池**部分中，转到要启用节点自动替换的节点池。在 **Recreate Unreachable After** 字段中，输入 0。 +1. 单击**保存**。 + +**结果**：已禁用节点池的节点自动替换。 + +### 云凭证 + +节点模板可以使用云凭证，来存储用于在云提供商中启动节点的凭证，其优点是： + +- 凭证会存储为更安全的 Kubernetes 密文，而且你无需每次都输入凭证便可编辑节点模板。 + +- 创建云凭证后，你可以重新使用该凭证来创建其他节点模板。 + +- 多个节点模板可以使用相同的云凭证来创建节点池。如果你的密钥被泄露或过期，则可以在一个位置更新云凭证，从而一次更新所有使用该凭证的节点模板。 + +创建云凭证后，用户可以[管理创建的云凭证](../../../../reference-guides/user-settings/manage-cloud-credentials.md)。 + +### 主机驱动 + +如果你找不到想要的主机驱动，你可以在 Rancher 的[内置主机驱动](../../authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md#激活停用主机驱动)中查看并激活它，也可以[添加自定义主机驱动](../../authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md#添加自定义主机驱动)。 + +## RKE2 集群 + +Rancher 2.6 支持直接使用 Rancher UI 配置 [RKE2](https://docs.rke2.io/) 集群。RKE2，也称为 RKE Government，是一个完全符合标准的 Kubernetes 发行版，它专注于安全性和合规性。 + +:::note + +对于 RKE2 集群模板，请参阅[此页面](../../manage-clusters/manage-cluster-templates.md#rke2-集群模板)了解更多信息。 + +::: + +### 节点角色 + +RKE2 CLI 公开了 `server` 和 `agent` 两个角色，它们分别代表 Kubernetes 节点角色 `etcd` + `controlplane` 和 `worker`。通过 Rancher 2.6 中的 RKE2 集成，RKE2 节点池可以分配更细粒度的角色，例如 `etcd` 和 `controlplane`。 + +你可以在 RKE2 CLI 中使用标志和节点污染，来控制调度工作负载和 Kubernetes master 节点的位置，从而使用 `etcd`，`controlplane` 和 `worker` 节点功能。这些角色没有在 RKE2 CLI 中实现为第一级角色的原因是，RKE2 被概念化为一组原始构建块，使用 Rancher 等编排系统得到最佳利用。 + +在 Rancher 中实现这三个节点角色，表示 Rancher 管理的 RKE2 集群能够轻松使用为 RKE 集群推荐的相同架构的所有最佳实践。 + +在[推荐的集群架构](../../kubernetes-clusters-in-rancher-setup/checklist-for-production-ready-clusters/recommended-cluster-architecture.md)中，我们概述了每个角色集群应该有多少节点： + +- 至少拥有三个角色为 etcd 的节点，来确保失去一个节点时仍能存活。 +- 至少两个节点具有 controlplane 角色，以实现主组件高可用性。 +- 至少两个具有 worker 角色的节点，用于在节点故障时重新安排工作负载。 diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md new file mode 100644 index 00000000000..396f6370108 --- /dev/null +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md @@ -0,0 +1,60 @@ +--- +title: 创建 vSphere 集群 +description: 使用 Rancher 创建 vSphere 集群。集群可能包括具有不同属性的 VM 组，这些属性可用于细粒度控制节点的大小。 +--- + + + + + +import YouTube from '@site/src/components/YouTube' + +你可以结合使用 Rancher 与 vSphere，从而在本地体验云环境的操作。 + +Rancher 可以在 vSphere 中配置节点并在其上安装 Kubernetes。在 vSphere 中创建 Kubernetes 集群时，Rancher 首先与 vCenter API 通信来配置指定数量的虚拟机。然后在它们之上安装 Kubernetes。 + +vSphere 集群可能由多组具有不同属性（例如内存或 vCPU 数量）的 VM 组成。这种分组允许对每个 Kubernetes 角色的节点大小进行细粒度控制。 + +## Rancher v2.3 中的 vSphere 增强功能 + +我们更新了 vSphere 节点模板，使你可以通过以下增强功能在本地体验云操作： + +### 自我修复的节点池 + +使用 Rancher 配置 vSphere 节点的最大优势之一，是允许你在本地集群中使用 Rancher 的自我修复节点池（也称为[节点自动替换功能](../use-new-nodes-in-an-infra-provider.md#节点自动替换)）。自我修复节点池的功能帮助你替换无状态应用的 worker 节点。当 Rancher 使用节点模板配置节点时，Rancher 可以自动替换无法访问的节点。 + +:::caution + +不建议在 master 节点或连接了持久卷的节点的节点池上启用节点自动替换，因为虚拟机会被临时处理。节点池中的节点与集群断开连接时，其持久卷将被破坏，从而导致有状态应用的数据丢失。 + +::: + +### 实例和调度的动态填充选项 + +vSphere 的节点模板已更新。当你使用 vSphere 凭证创建节点模板时，该模板会自动填充你在 vSphere 控制台中可以访问的相同的虚拟机配置选项。 + +要填充的字段设置需要满足[先决条件](provision-kubernetes-clusters-in-vsphere.md#vsphere-中的准备工作)。 + +### 更多支持的操作系统 + +你可以使用任何支持 `cloud-init` 的操作系统来配置 VM。[cloud config](https://cloudinit.readthedocs.io/en/latest/topics/examples.html) 仅支持 YAML 格式。 + +### v2.3.3 节点模板功能的视频介绍 + +在这段 YouTube 视频中，我们演示了如何使用新的节点模板在本地环境体验云环境一样的操作。 + + + +## 创建 vSphere 集群 + +在[本节](provision-kubernetes-clusters-in-vsphere.md)中，你将学习如何使用 Rancher 在 vSphere 中安装 [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes 集群。 + +## 配置存储 + +有关如何使用 Rancher 在 vSphere 中配置存储的示例，请参阅[本节](../../../manage-clusters/provisioning-storage-examples/vsphere-storage.md)。要在 vSphere 中动态配置存储，你必须启用 vSphere 云提供商。请参阅[树内 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md)和[树外 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 + +## 启用 vSphere 云提供商 + +在 Rancher 中设置云提供商时，Rancher Server 可以自动为集群配置新的基础设施，包括新节点或持久存储设备。 + +有关启用 vSphere 云提供商的详细信息，请参阅[树内 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md)和[树外 vSphere 配置](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md)。 From 84d4214bb22ceac3119a4537428bef167b991ac0 Mon Sep 17 00:00:00 2001 From: Lucas Saintarbor Date: Mon, 6 May 2024 09:48:56 -0700 Subject: [PATCH 64/67] Add formatting for tables (#1259) * Add formatting for tables * Remove min-width: 150px for first column --- src/css/custom.css | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/css/custom.css b/src/css/custom.css index 7172cf684c5..2ac4b3638a1 100644 --- a/src/css/custom.css +++ b/src/css/custom.css @@ -65,6 +65,15 @@ @apply font-sans antialiased; } + table td:nth-child(1) { + overflow-wrap: normal; + } + + td { + width: auto; + overflow-wrap: anywhere; /* For browser compatibility */ + } + .homepage a { --ifm-link-hover-color: currentColor; } From 31c279bd42a450e8ac61cf83c3837f76ce651587 Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Mon, 6 May 2024 15:12:01 -0400 Subject: [PATCH 65/67] #1240 Adding 'VMware' to mentions of vSphere in titles/headings (#1261) * Adding 'VMware' to mentions of vSphere in titles/headings * fix bad links * vSphere stragglers * versioning --- .../migrate-to-out-of-tree-vsphere.md | 4 ++-- .../configure-in-tree-vsphere.md | 4 ++-- .../configure-out-of-tree-vsphere.md | 6 +++--- .../set-up-cloud-providers.md | 2 +- .../use-windows-clusters/use-windows-clusters.md | 2 +- .../vsphere/create-a-vm-template.md | 2 +- .../vsphere/create-credentials.md | 4 ++-- .../provision-kubernetes-clusters-in-vsphere.md | 16 ++++++++-------- .../vsphere/vsphere.md | 14 +++++++------- .../create-kubernetes-persistent-storage.md | 2 +- .../vsphere-storage.md | 6 +++--- .../rancher-managed-clusters-in-vsphere.md | 2 +- .../rancher-managed-clusters.md | 2 +- .../on-premises-rancher-in-vsphere.md | 4 ++-- .../rancher-server/rancher-server.md | 2 +- .../node-template-configuration/vsphere.md | 2 +- .../version-2.0-2.4.json | 2 +- sidebars.js | 2 +- .../vsphere-storage.md | 4 ++-- .../vsphere/create-credentials.md | 4 ++-- .../provision-kubernetes-clusters-in-vsphere.md | 16 ++++++++-------- .../vsphere/vsphere.md | 14 +++++++------- .../vsphere/prior-to-v2.0.4.md | 2 +- .../vsphere/v2.0.4.md | 2 +- .../vsphere/v2.2.0.md | 2 +- .../vsphere/v2.3.0.md | 2 +- .../vsphere/v2.3.3.md | 2 +- .../vsphere/vsphere.md | 2 +- .../create-kubernetes-persistent-storage.md | 2 +- .../vsphere-storage.md | 4 ++-- .../vsphere/configure-in-tree-vsphere.md | 4 ++-- .../vsphere/configure-out-of-tree-vsphere.md | 2 +- .../vsphere/create-credentials.md | 4 ++-- .../provision-kubernetes-clusters-in-vsphere.md | 16 ++++++++-------- .../vsphere/vsphere.md | 16 ++++++++-------- .../use-windows-clusters/use-windows-clusters.md | 2 +- .../rancher-managed-clusters-in-vsphere.md | 2 +- .../rancher-managed-clusters.md | 2 +- .../on-premises-rancher-in-vsphere.md | 4 ++-- .../rancher-server/rancher-server.md | 2 +- .../node-template-configuration/vsphere.md | 2 +- .../configure-in-tree-vsphere.md | 4 ++-- .../configure-out-of-tree-vsphere.md | 6 +++--- .../use-windows-clusters/use-windows-clusters.md | 2 +- .../vsphere/create-a-vm-template.md | 2 +- .../vsphere/create-credentials.md | 4 ++-- .../provision-kubernetes-clusters-in-vsphere.md | 16 ++++++++-------- .../vsphere/vsphere.md | 14 +++++++------- .../create-kubernetes-persistent-storage.md | 2 +- .../vsphere-storage.md | 6 +++--- .../rancher-managed-clusters-in-vsphere.md | 2 +- .../rancher-managed-clusters.md | 2 +- .../on-premises-rancher-in-vsphere.md | 4 ++-- .../rancher-server/rancher-server.md | 2 +- .../node-template-configuration/vsphere.md | 2 +- .../migrate-to-out-of-tree-vsphere.md | 4 ++-- .../configure-in-tree-vsphere.md | 4 ++-- .../configure-out-of-tree-vsphere.md | 6 +++--- .../use-windows-clusters/use-windows-clusters.md | 2 +- .../vsphere/create-a-vm-template.md | 2 +- .../vsphere/create-credentials.md | 4 ++-- .../provision-kubernetes-clusters-in-vsphere.md | 16 ++++++++-------- .../vsphere/vsphere.md | 14 +++++++------- .../create-kubernetes-persistent-storage.md | 2 +- .../vsphere-storage.md | 6 +++--- .../rancher-managed-clusters-in-vsphere.md | 2 +- .../rancher-managed-clusters.md | 2 +- .../on-premises-rancher-in-vsphere.md | 4 ++-- .../rancher-server/rancher-server.md | 2 +- .../node-template-configuration/vsphere.md | 2 +- .../migrate-to-out-of-tree-vsphere.md | 4 ++-- .../configure-in-tree-vsphere.md | 4 ++-- .../configure-out-of-tree-vsphere.md | 6 +++--- .../use-windows-clusters/use-windows-clusters.md | 2 +- .../vsphere/create-a-vm-template.md | 2 +- .../vsphere/create-credentials.md | 4 ++-- .../provision-kubernetes-clusters-in-vsphere.md | 16 ++++++++-------- .../vsphere/vsphere.md | 14 +++++++------- .../create-kubernetes-persistent-storage.md | 2 +- .../vsphere-storage.md | 6 +++--- .../rancher-managed-clusters-in-vsphere.md | 2 +- .../rancher-managed-clusters.md | 2 +- .../on-premises-rancher-in-vsphere.md | 4 ++-- .../rancher-server/rancher-server.md | 2 +- .../node-template-configuration/vsphere.md | 2 +- versioned_sidebars/version-2.0-2.4-sidebars.json | 4 ++-- versioned_sidebars/version-2.5-sidebars.json | 4 ++-- versioned_sidebars/version-2.6-sidebars.json | 2 +- versioned_sidebars/version-2.7-sidebars.json | 2 +- versioned_sidebars/version-2.8-sidebars.json | 2 +- 90 files changed, 208 insertions(+), 208 deletions(-) diff --git a/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md b/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md index c93da30901c..df230919675 100644 --- a/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md +++ b/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md @@ -1,12 +1,12 @@ --- -title: Migrating vSphere In-tree to Out-of-tree +title: Migrating VMware vSphere In-tree to Out-of-tree --- -Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. +Kubernetes is moving away from maintaining cloud providers in-tree. VMware vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. This page covers how to migrate from the in-tree vSphere cloud provider to out-of-tree, and manage the existing VMs post migration. diff --git a/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md b/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md index 7168b4ab493..90f326de2d1 100644 --- a/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md +++ b/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md @@ -1,12 +1,12 @@ --- -title: Setting Up an In-tree vSphere Cloud Provider +title: Setting Up an In-tree VMware vSphere Cloud Provider --- -To set up the in-tree vSphere cloud provider, follow these steps while creating the vSphere cluster in Rancher: +To set up the in-tree VMware vSphere cloud provider, follow these steps while creating the vSphere cluster in Rancher: 1. Set **Cloud Provider** option to `Custom` or `Custom (In-Tree)`. 1. Click on **Edit as YAML** diff --git a/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md b/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md index 43d581ab2a3..6e866abc971 100644 --- a/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md +++ b/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md @@ -1,12 +1,12 @@ --- -title: Setting Up an Out-of-tree vSphere Cloud Provider +title: Setting Up an Out-of-tree VMware vSphere Cloud Provider --- -Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. +Kubernetes is moving away from maintaining cloud providers in-tree. VMware vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. This page covers how to install the Cloud Provider Interface (CPI) and Cloud Storage Interface (CSI) plugins after setting up a cluster. @@ -25,7 +25,7 @@ Using the vSphere out-of-tree cloud provider requires Linux nodes and is not sup The Cloud Provider Interface (CPI) should be installed first before installing the Cloud Storage Interface (CSI). -### 1. Create a vSphere cluster +### 1. Create a VMware vSphere cluster 1. Click **☰ > Cluster Management**. 1. On the **Clusters** page, click **Create**. diff --git a/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md b/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md index d6c9834b666..4cf5e31ad8c 100644 --- a/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md +++ b/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/set-up-cloud-providers.md @@ -33,7 +33,7 @@ For details on enabling the Azure cloud provider, refer to [this page.](azure.md For details on enabling the Google Compute Engine cloud provider, refer to [this page.](google-compute-engine.md) -### Setting up the vSphere Cloud Provider +### Setting up the VMware vSphere Cloud Provider For details on enabling the vSphere cloud provider, refer to [in-tree vSphere config](configure-in-tree-vsphere.md) and [out-of-tree vSphere config](configure-out-of-tree-vsphere.md). diff --git a/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md b/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md index 4c5f811c5e6..4c796540c2e 100644 --- a/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md +++ b/docs/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md @@ -86,7 +86,7 @@ Some Linux operating systems accept multiple domain names separated by spaces. H ::: -### Rancher on vSphere with ESXi 6.7u2 and above +### Rancher on VMware vSphere with ESXi 6.7u2 and above If you are using Rancher on VMware vSphere with ESXi 6.7u2 or later with Red Hat Enterprise Linux 8.3, CentOS 8.3, or SUSE Enterprise Linux 15 SP2 or later, it is necessary to disable the `vmxnet3` virtual network adapter hardware offloading feature. Failure to do so will result in all network connections between pods on different cluster nodes to fail with timeout errors. All connections from Windows pods to critical services running on Linux nodes, such as CoreDNS, will fail as well. It is also possible that external connections may fail. This issue is the result of Linux distributions enabling the hardware offloading feature in `vmxnet3` and a bug in the `vmxnet3` hardware offloading feature that results in the discarding of packets for guest overlay traffic. To address this issue, it is necessary disable the `vmxnet3` hardware offloading feature. This setting does not survive reboot, so it is necessary to disable on every boot. The recommended course of action is to create a systemd unit file at `/etc/systemd/system/disable_hw_offloading.service`, which disables the `vmxnet3` hardware offloading feature on boot. A sample systemd unit file which disables the `vmxnet3` hardware offloading feature is as follows. Note that `` must be customized to the host `vmxnet3` network interface, e.g., `ens192`: diff --git a/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md b/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md index cfdd8da1f19..55e23a9235a 100644 --- a/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md +++ b/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md @@ -1,5 +1,5 @@ --- -title: Creating a vSphere Virtual Machine Template +title: Creating a VMware vSphere Virtual Machine Template --- diff --git a/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md b/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md index d228c27c1e7..20eb639ad31 100644 --- a/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md +++ b/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md @@ -1,12 +1,12 @@ --- -title: Creating Credentials in the vSphere Console +title: Creating Credentials in the VMware vSphere Console --- -This section describes how to create a vSphere username and password. You will need to provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. +This section describes how to create a VMware vSphere username and password. You must provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. The following table lists the permissions required for the vSphere user account: diff --git a/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md b/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md index 9aec21efdf6..ae1bcf2fcb2 100644 --- a/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md +++ b/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md @@ -1,12 +1,12 @@ --- -title: Provisioning Kubernetes Clusters in vSphere +title: Provisioning Kubernetes Clusters in VMware vSphere --- -In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in vSphere. +In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in VMware vSphere. First, you will set up your vSphere cloud credentials in Rancher. Then you will use your cloud credentials to create a node template, which Rancher will use to provision nodes in vSphere. @@ -17,16 +17,16 @@ For details on configuring the vSphere node template, refer to the [vSphere node For details on configuring RKE Kubernetes clusters in Rancher, refer to the [cluster configuration reference.](../../../../../reference-guides/cluster-configuration/rancher-server-configuration/rke1-cluster-configuration.md#rke-cluster-config-file-reference) -- [Preparation in vSphere](#preparation-in-vsphere) -- [Creating a vSphere Cluster](#creating-a-vsphere-cluster) +- [Preparation in vSphere](#preparation-in-vmware-vsphere) +- [Creating a vSphere Cluster](#creating-a-vmware-vsphere-cluster) -## Preparation in vSphere +## Preparation in VMware vSphere This section describes the requirements for setting up vSphere so that Rancher can provision VMs and clusters. The node templates are documented and tested with the vSphere Web Services API version 6.5. -### Create Credentials in vSphere +### Create Credentials in VMware vSphere Before proceeding to create a cluster, you must ensure that you have a vSphere user with sufficient permissions. When you set up a node template, the template will need to use these vSphere credentials. @@ -42,7 +42,7 @@ It must be ensured that the hosts running the Rancher server are able to establi See [Node Networking Requirements](../../../kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md#networking-requirements) for a detailed list of port requirements applicable for creating nodes on an infrastructure provider. -### Valid ESXi License for vSphere API Access +### Valid ESXi License for VMware vSphere API Access The free ESXi license does not support API access. The vSphere servers must have a valid or evaluation ESXi license. @@ -50,7 +50,7 @@ The free ESXi license does not support API access. The vSphere servers must have If you have a cluster with DRS enabled, setting up [VM-VM Affinity Rules](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.resmgmt.doc/GUID-7297C302-378F-4AF2-9BD6-6EDB1E0A850A.html) is recommended. These rules allow VMs assigned the etcd and control-plane roles to operate on separate ESXi hosts when they are assigned to different node pools. This practice ensures that the failure of a single physical machine does not affect the availability of those planes. -## Creating a vSphere Cluster +## Creating a VMware vSphere Cluster 1. [Create your cloud credentials](#1-create-your-cloud-credentials) 2. [Create a node template with your cloud credentials](#2-create-a-node-template-with-your-cloud-credentials) diff --git a/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md b/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md index e1d63847e59..ed7dbb7cd35 100644 --- a/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md +++ b/docs/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md @@ -1,6 +1,6 @@ --- -title: Creating a vSphere Cluster -description: Use Rancher to create a vSphere cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. +title: Creating a VMware vSphere Cluster +description: Use Rancher to create a VMware vSphere cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. --- @@ -9,13 +9,13 @@ description: Use Rancher to create a vSphere cluster. It may consist of groups o import YouTube from '@site/src/components/YouTube' -By using Rancher with vSphere, you can bring cloud operations on-premises. +By using Rancher with VMware vSphere, you can bring cloud operations on-premises. Rancher can provision nodes in vSphere and install Kubernetes on them. When creating a Kubernetes cluster in vSphere, Rancher first provisions the specified number of virtual machines by communicating with the vCenter API. Then it installs Kubernetes on top of them. A vSphere cluster may consist of multiple groups of VMs with distinct properties, such as the amount of memory or the number of vCPUs. This grouping allows for fine-grained control over the sizing of nodes for each Kubernetes role. -## vSphere Enhancements in Rancher v2.3 +## VMware vSphere Enhancements in Rancher v2.3 The vSphere node templates have been updated, allowing you to bring cloud operations on-premises with the following enhancements: @@ -33,7 +33,7 @@ It is not recommended to enable node auto-replace on a node pool of master nodes Node templates for vSphere have been updated so that when you create a node template with your vSphere credentials, the template is automatically populated with the same options for provisioning VMs that you have access to in the vSphere console. -For the fields to be populated, your setup needs to fulfill the [prerequisites.](provision-kubernetes-clusters-in-vsphere.md#preparation-in-vsphere) +For the fields to be populated, your setup needs to fulfill the [prerequisites.](provision-kubernetes-clusters-in-vsphere.md#preparation-in-vmware-vsphere) ### More Supported Operating Systems @@ -45,7 +45,7 @@ In this YouTube video, we demonstrate how to set up a node template with the new -## Creating a vSphere Cluster +## Creating a VMware vSphere Cluster In [this section,](provision-kubernetes-clusters-in-vsphere.md) you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in vSphere. @@ -53,7 +53,7 @@ In [this section,](provision-kubernetes-clusters-in-vsphere.md) you'll learn how For an example of how to provision storage in vSphere using Rancher, refer to [this section.](../../../manage-clusters/provisioning-storage-examples/vsphere-storage.md) In order to dynamically provision storage in vSphere, the vSphere provider must be enabled. Refer to [in-tree vSphere config](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md) and [out-of-tree vSphere config](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md). -## Enabling the vSphere Cloud Provider +## Enabling the VMware vSphere Cloud Provider When a cloud provider is set up in Rancher, the Rancher server can automatically provision new infrastructure for the cluster, including new nodes or persistent storage devices. diff --git a/docs/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/docs/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md index 7ada9cd85af..67a8c9dffbd 100644 --- a/docs/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md +++ b/docs/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -67,7 +67,7 @@ In [Rancher Launched Kubernetes clusters](../../launch-kubernetes-with-rancher/l ### hostPath Volumes Before you create a hostPath volume, you need to set up an [extra_bind](https://rancher.com/docs/rke/latest/en/config-options/services/services-extras/#extra-binds/) in your cluster configuration. This will mount the path as a volume in your kubelets, which can then be used for hostPath volumes in your workloads. -### Migrating vSphere Cloud Provider from In-tree to Out-of-tree +### Migrating VMware vSphere Cloud Provider from In-tree to Out-of-tree Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. diff --git a/docs/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md b/docs/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md index 0bede8b10ae..9d818629963 100644 --- a/docs/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md +++ b/docs/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md @@ -1,12 +1,12 @@ --- -title: vSphere Storage +title: VMware vSphere Storage --- -To provide stateful workloads with vSphere storage, we recommend creating a vSphereVolume StorageClass. This practice dynamically provisions vSphere storage when workloads request volumes through a PersistentVolumeClaim. +To provide stateful workloads with VMware vSphere storage, we recommend creating a vSphereVolume StorageClass. This practice dynamically provisions vSphere storage when workloads request volumes through a PersistentVolumeClaim. In order to dynamically provision storage in vSphere, the vSphere provider must be enabled. See the following pages for more: [Out-of-tree vSphere](../../../new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md) and [in-tree vSphere](../../../new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md). @@ -35,7 +35,7 @@ The following steps can also be performed using the `kubectl` command line tool. 5. Optionally, specify additional properties for this storage class under **Parameters**. Refer to the [vSphere storage documentation](https://github.com/vmware-archive/vsphere-storage-for-kubernetes/blob/master/documentation/storageclass.md) for details. 5. Click **Create**. -### Creating a Workload with a vSphere Volume +### Creating a Workload with a VMware vSphere Volume 1. In the left navigation bar, click **Workload**. 1. Click **Create**. diff --git a/docs/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md b/docs/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md index 9d24fb3c5db..73242a90abb 100644 --- a/docs/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md +++ b/docs/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md @@ -1,5 +1,5 @@ --- -title: Best Practices for Rancher Managed vSphere Clusters +title: Best Practices for Rancher Managed VMware vSphere Clusters --- diff --git a/docs/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md b/docs/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md index ea4697f8e76..1e73af430af 100644 --- a/docs/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md +++ b/docs/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md @@ -18,6 +18,6 @@ Configuring sensible monitoring and alerting rules is vital for running any prod Running well-built containers can greatly impact the overall performance and security of your environment. Refer to this [guide](tips-to-set-up-containers.md) for tips. -### Best Practices for Rancher Managed vSphere Clusters +### Best Practices for Rancher Managed VMware vSphere Clusters This [guide](rancher-managed-clusters-in-vsphere.md) outlines a reference architecture for provisioning downstream Rancher clusters in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. diff --git a/docs/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md b/docs/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md index 6d35b2dc2d8..b5b412a1bef 100644 --- a/docs/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md +++ b/docs/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md @@ -1,12 +1,12 @@ --- -title: Installing Rancher in a vSphere Environment +title: Installing Rancher in a VMware vSphere Environment --- -This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. +This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a VMware vSphere environment. It also desctibes standard vSphere best practices as documented by VMware.
Solution Overview
diff --git a/docs/reference-guides/best-practices/rancher-server/rancher-server.md b/docs/reference-guides/best-practices/rancher-server/rancher-server.md index 8e24e145482..8babb9636fd 100644 --- a/docs/reference-guides/best-practices/rancher-server/rancher-server.md +++ b/docs/reference-guides/best-practices/rancher-server/rancher-server.md @@ -16,6 +16,6 @@ Refer to this [guide](tips-for-running-rancher.md) for our general advice for se This [guide](rancher-deployment-strategy.md) is designed to help you choose whether a regional deployment strategy or a hub-and-spoke deployment strategy is better for a Rancher server that manages downstream Kubernetes clusters. -### Installing Rancher in a vSphere Environment +### Installing Rancher in a VMware vSphere Environment This [guide](on-premises-rancher-in-vsphere.md) outlines a reference architecture for installing Rancher in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. \ No newline at end of file diff --git a/docs/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md b/docs/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md index ace79a39ca7..cfc2ed183b7 100644 --- a/docs/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md +++ b/docs/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md @@ -1,5 +1,5 @@ --- -title: VSphere Node Template Configuration +title: VMware vSphere Node Template Configuration --- diff --git a/i18n/zh/docusaurus-plugin-content-docs/version-2.0-2.4.json b/i18n/zh/docusaurus-plugin-content-docs/version-2.0-2.4.json index e35e1b3f6e7..4d0f36dd3cb 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/version-2.0-2.4.json +++ b/i18n/zh/docusaurus-plugin-content-docs/version-2.0-2.4.json @@ -308,7 +308,7 @@ "description": "The label for category Node Template Configuration in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Creating a vSphere Cluster": { - "message": "Creating a vSphere Cluster", + "message": "Creating a VMware vSphere Cluster", "description": "The label for category Creating a vSphere Cluster in sidebar tutorialSidebar" }, "sidebar.tutorialSidebar.category.Single-Node Rancher in Docker": { diff --git a/sidebars.js b/sidebars.js index b432826abd6..5eb8795b0a1 100644 --- a/sidebars.js +++ b/sidebars.js @@ -534,7 +534,7 @@ const sidebars = { "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster", { type: 'category', - label: 'Creating a vSphere Cluster', + label: 'Creating a VMware vSphere Cluster', link: { type: 'doc', id: "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere", diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/provisioning-storage-examples/vsphere-storage.md b/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/provisioning-storage-examples/vsphere-storage.md index 6dc96b359c2..abdbee96b96 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/provisioning-storage-examples/vsphere-storage.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/provisioning-storage-examples/vsphere-storage.md @@ -1,5 +1,5 @@ --- -title: vSphere Storage +title: VMware vSphere Storage --- @@ -31,7 +31,7 @@ In order to provision vSphere volumes in a cluster created with the [Rancher Kub 5. Optionally, specify additional properties for this storage class under **Parameters**. Refer to the [vSphere storage documentation](https://github.com/vmware-archive/vsphere-storage-for-kubernetes/blob/master/documentation/storageclass.md) for details. 5. Click **Save**. -### Creating a Workload with a vSphere Volume +### Creating a Workload with a VMware vSphere Volume 1. From the cluster where you configured vSphere storage, begin creating a workload as you would in [Deploying Workloads](../../../../new-user-guides/kubernetes-resources-setup/workloads-and-pods/deploy-workloads.md). 2. For **Workload Type**, select **Stateful set of 1 pod**. diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md index a7f1f7f6d6b..31da9a9cd7e 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md @@ -1,12 +1,12 @@ --- -title: Creating Credentials in the vSphere Console +title: Creating Credentials in the VMware vSphere Console --- -This section describes how to create a vSphere username and password. You will need to provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. +This section describes how to create a VMware vSphere username and password. You must provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. The following table lists the permissions required for the vSphere user account: diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md index fc7212d9a47..13bc1cfd6f4 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md @@ -1,12 +1,12 @@ --- -title: Provisioning Kubernetes Clusters in vSphere +title: Provisioning Kubernetes Clusters in VMware vSphere --- -In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in vSphere. +In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in VMware vSphere. First, you will set up your vSphere cloud credentials in Rancher. Then you will use your cloud credentials to create a node template, which Rancher will use to provision nodes in vSphere. @@ -16,16 +16,16 @@ For details on configuring the vSphere node template, refer to the [vSphere node For details on configuring RKE Kubernetes clusters in Rancher, refer to the [cluster configuration reference.](../../../../../../reference-guides/cluster-configuration/rancher-server-configuration/rke1-cluster-configuration.md) -- [Preparation in vSphere](#preparation-in-vsphere) -- [Creating a vSphere Cluster](#creating-a-vsphere-cluster) +- [Preparation in vSphere](#preparation-in-vmware-vsphere) +- [Creating a vSphere Cluster](#creating-a-vmware-vsphere-cluster) -## Preparation in vSphere +## Preparation in VMware vSphere This section describes the requirements for setting up vSphere so that Rancher can provision VMs and clusters. The node templates are documented and tested with the vSphere Web Services API version 6.5. -### Create Credentials in vSphere +### Create Credentials in VMware vSphere Before proceeding to create a cluster, you must ensure that you have a vSphere user with sufficient permissions. When you set up a node template, the template will need to use these vSphere credentials. @@ -41,7 +41,7 @@ It must be ensured that the hosts running the Rancher server are able to establi See [Node Networking Requirements](../../../node-requirements-for-rancher-managed-clusters.md#networking-requirements) for a detailed list of port requirements applicable for creating nodes on an infrastructure provider. -### Valid ESXi License for vSphere API Access +### Valid ESXi License for VMware vSphere API Access The free ESXi license does not support API access. The vSphere servers must have a valid or evaluation ESXi license. @@ -49,7 +49,7 @@ The free ESXi license does not support API access. The vSphere servers must have If you have a cluster with DRS enabled, setting up [VM-VM Affinity Rules](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.resmgmt.doc/GUID-7297C302-378F-4AF2-9BD6-6EDB1E0A850A.html) is recommended. These rules allow VMs assigned the etcd and control-plane roles to operate on separate ESXi hosts when they are assigned to different node pools. This practice ensures that the failure of a single physical machine does not affect the availability of those planes. -## Creating a vSphere Cluster +## Creating a VMware vSphere Cluster diff --git a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md index 4314460ead5..df349d4476b 100644 --- a/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md +++ b/versioned_docs/version-2.0-2.4/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md @@ -1,6 +1,6 @@ --- -title: Creating a vSphere Cluster -description: Use Rancher to create a vSphere cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. +title: Creating a VMware vSphere Cluster +description: Use Rancher to create a VMware vSphere cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. --- @@ -9,13 +9,13 @@ description: Use Rancher to create a vSphere cluster. It may consist of groups o import YouTube from '@site/src/components/YouTube' -By using Rancher with vSphere, you can bring cloud operations on-premises. +By using Rancher with VMware vSphere, you can bring cloud operations on-premises. Rancher can provision nodes in vSphere and install Kubernetes on them. When creating a Kubernetes cluster in vSphere, Rancher first provisions the specified number of virtual machines by communicating with the vCenter API. Then it installs Kubernetes on top of them. A vSphere cluster may consist of multiple groups of VMs with distinct properties, such as the amount of memory or the number of vCPUs. This grouping allows for fine-grained control over the sizing of nodes for each Kubernetes role. -## vSphere Enhancements in Rancher v2.3 +## VMware vSphere Enhancements in Rancher v2.3 The vSphere node templates have been updated, allowing you to bring cloud operations on-premises with the following enhancements: @@ -33,7 +33,7 @@ _Available as of v2.3.3_ Node templates for vSphere have been updated so that when you create a node template with your vSphere credentials, the template is automatically populated with the same options for provisioning VMs that you have access to in the vSphere console. -For the fields to be populated, your setup needs to fulfill the [prerequisites.](provision-kubernetes-clusters-in-vsphere.md#preparation-in-vsphere) +For the fields to be populated, your setup needs to fulfill the [prerequisites.](provision-kubernetes-clusters-in-vsphere.md#preparation-in-vmware-vsphere) ### More Supported Operating Systems @@ -47,7 +47,7 @@ In this YouTube video, we demonstrate how to set up a node template with the new -## Creating a vSphere Cluster +## Creating a VMware vSphere Cluster In [this section,](provision-kubernetes-clusters-in-vsphere.md) you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in vSphere. @@ -55,7 +55,7 @@ In [this section,](provision-kubernetes-clusters-in-vsphere.md) you'll learn how For an example of how to provision storage in vSphere using Rancher, refer to [this section.](../../../../../advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/provisioning-storage-examples/vsphere-storage.md) In order to dynamically provision storage in vSphere, the vSphere provider must be [enabled.](../../set-up-cloud-providers/other-cloud-providers/vsphere.md) -## Enabling the vSphere Cloud Provider +## Enabling the VMware vSphere Cloud Provider When a cloud provider is set up in Rancher, the Rancher server can automatically provision new infrastructure for the cluster, including new nodes or persistent storage devices. diff --git a/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/prior-to-v2.0.4.md b/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/prior-to-v2.0.4.md index 155a2d452e5..a6fef9dc901 100644 --- a/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/prior-to-v2.0.4.md +++ b/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/prior-to-v2.0.4.md @@ -1,5 +1,5 @@ --- -title: vSphere Node Template Configuration in Rancher before v2.0.4 +title: VMware vSphere Node Template Configuration in Rancher before v2.0.4 --- ## Account Access diff --git a/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.0.4.md b/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.0.4.md index e065f0b050d..2260c5c9883 100644 --- a/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.0.4.md +++ b/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.0.4.md @@ -1,5 +1,5 @@ --- -title: vSphere Node Template Configuration in Rancher v2.0.4 +title: VMware vSphere Node Template Configuration in Rancher v2.0.4 --- ## Account Access diff --git a/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.2.0.md b/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.2.0.md index c27ba591499..f4db379de94 100644 --- a/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.2.0.md +++ b/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.2.0.md @@ -1,5 +1,5 @@ --- -title: vSphere Node Template Configuration in Rancher v2.2.0 +title: VMware vSphere Node Template Configuration in Rancher v2.2.0 --- ## Account Access diff --git a/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.3.0.md b/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.3.0.md index d086a7fce34..92b12cffe50 100644 --- a/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.3.0.md +++ b/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.3.0.md @@ -1,5 +1,5 @@ --- -title: vSphere Node Template Configuration in Rancher v2.3.0 +title: VMware vSphere Node Template Configuration in Rancher v2.3.0 --- ## Account Access diff --git a/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.3.3.md b/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.3.3.md index 2447d41448a..75e5da83426 100644 --- a/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.3.3.md +++ b/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/v2.3.3.md @@ -1,5 +1,5 @@ --- -title: vSphere Node Template Configuration in Rancher v2.3.3 +title: VMware vSphere Node Template Configuration in Rancher v2.3.3 --- ## Account Access diff --git a/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/vsphere.md b/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/vsphere.md index 2c169d0c9d8..47c91fb883a 100644 --- a/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/vsphere.md +++ b/versioned_docs/version-2.0-2.4/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/vsphere.md @@ -1,5 +1,5 @@ --- -title: VSphere Node Template Configuration +title: VMware vSphere Node Template Configuration --- diff --git a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md index 847e2eed848..495c6cd3223 100644 --- a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md +++ b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -67,7 +67,7 @@ In [Rancher Launched Kubernetes clusters](../../../new-user-guides/kubernetes-cl ### hostPath Volumes Before you create a hostPath volume, you need to set up an [extra_bind](https://rancher.com/docs/rke/latest/en/config-options/services/services-extras/#extra-binds/) in your cluster configuration. This will mount the path as a volume in your kubelets, which can then be used for hostPath volumes in your workloads. -### Migrating vSphere Cloud Provider from In-tree to Out-of-tree +### Migrating VMware vSphere Cloud Provider from In-tree to Out-of-tree Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. diff --git a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/provisioning-storage-examples/vsphere-storage.md b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/provisioning-storage-examples/vsphere-storage.md index 3639768ee6b..60b3bc9d589 100644 --- a/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/provisioning-storage-examples/vsphere-storage.md +++ b/versioned_docs/version-2.5/how-to-guides/advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/provisioning-storage-examples/vsphere-storage.md @@ -1,5 +1,5 @@ --- -title: vSphere Storage +title: VMware vSphere Storage --- @@ -31,7 +31,7 @@ In order to provision vSphere volumes in a cluster created with the [Rancher Kub 5. Optionally, specify additional properties for this storage class under **Parameters**. Refer to the [vSphere storage documentation](https://github.com/vmware-archive/vsphere-storage-for-kubernetes/blob/master/documentation/storageclass.md) for details. 5. Click **Save**. -### Creating a Workload with a vSphere Volume +### Creating a Workload with a VMware vSphere Volume 1. From the cluster where you configured vSphere storage, begin creating a workload as you would in [Deploying Workloads](../../../../new-user-guides/kubernetes-resources-setup/workloads-and-pods/deploy-workloads.md). 2. For **Workload Type**, select **Stateful set of 1 pod**. diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/set-up-cloud-providers/vsphere/configure-in-tree-vsphere.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/set-up-cloud-providers/vsphere/configure-in-tree-vsphere.md index aed0fa78f38..87c306f38f5 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/set-up-cloud-providers/vsphere/configure-in-tree-vsphere.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/set-up-cloud-providers/vsphere/configure-in-tree-vsphere.md @@ -1,12 +1,12 @@ --- -title: Configuring an In-tree vSphere Cloud Provider +title: Configuring an In-tree VMware vSphere Cloud Provider --- -To set up the in-tree vSphere cloud provider, follow these steps while creating the vSphere cluster in Rancher: +To set up the in-tree VMware vSphere cloud provider, follow these steps while creating the vSphere cluster in Rancher: 1. Set **Cloud Provider** option to `Custom` or `Custom (In-Tree)`. diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/set-up-cloud-providers/vsphere/configure-out-of-tree-vsphere.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/set-up-cloud-providers/vsphere/configure-out-of-tree-vsphere.md index dd7fc07c534..635bef23517 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/set-up-cloud-providers/vsphere/configure-out-of-tree-vsphere.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/set-up-cloud-providers/vsphere/configure-out-of-tree-vsphere.md @@ -27,7 +27,7 @@ Using the vSphere out-of-tree cloud provider requires Linux nodes and is not sup The Cloud Provider Interface (CPI) should be installed first before installing the Cloud Storage Interface (CSI). -### 1. Create a vSphere cluster +### 1. Create a VMware vSphere cluster 1. On the Clusters page, click on **Add Cluster** and select the **vSphere** option or **Existing Nodes** option. 1. Under **Cluster Options** in the **Cloud Provider** section, select **External (Out-of-tree)**. This sets the cloud provider option on the Kubernetes cluster to `external` which sets your Kubernetes cluster up to be configured with an out-of-tree cloud provider. diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md index a7f1f7f6d6b..31da9a9cd7e 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md @@ -1,12 +1,12 @@ --- -title: Creating Credentials in the vSphere Console +title: Creating Credentials in the VMware vSphere Console --- -This section describes how to create a vSphere username and password. You will need to provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. +This section describes how to create a VMware vSphere username and password. You must provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. The following table lists the permissions required for the vSphere user account: diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md index 9292eca320a..c733779aecc 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md @@ -1,12 +1,12 @@ --- -title: Provisioning Kubernetes Clusters in vSphere +title: Provisioning Kubernetes Clusters in VMware vSphere --- -In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in vSphere. +In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in VMware vSphere. First, you will set up your vSphere cloud credentials in Rancher. Then you will use your cloud credentials to create a node template, which Rancher will use to provision nodes in vSphere. @@ -16,16 +16,16 @@ For details on configuring the vSphere node template, refer to the [vSphere node For details on configuring RKE Kubernetes clusters in Rancher, refer to the [cluster configuration reference.](../../../../../../reference-guides/cluster-configuration/rancher-server-configuration/rke1-cluster-configuration.md) -- [Preparation in vSphere](#preparation-in-vsphere) -- [Creating a vSphere Cluster](#creating-a-vsphere-cluster) +- [Preparation in vSphere](#preparation-in-vmware-vsphere) +- [Creating a vSphere Cluster](#creating-a-vmware-vsphere-cluster) -## Preparation in vSphere +## Preparation in VMware vSphere This section describes the requirements for setting up vSphere so that Rancher can provision VMs and clusters. The node templates are documented and tested with the vSphere Web Services API version 6.5. -### Create Credentials in vSphere +### Create Credentials in VMware vSphere Before proceeding to create a cluster, you must ensure that you have a vSphere user with sufficient permissions. When you set up a node template, the template will need to use these vSphere credentials. @@ -41,7 +41,7 @@ It must be ensured that the hosts running the Rancher server are able to establi See [Node Networking Requirements](../../../node-requirements-for-rancher-managed-clusters.md#networking-requirements) for a detailed list of port requirements applicable for creating nodes on an infrastructure provider. -### Valid ESXi License for vSphere API Access +### Valid ESXi License for VMware vSphere API Access The free ESXi license does not support API access. The vSphere servers must have a valid or evaluation ESXi license. @@ -49,7 +49,7 @@ The free ESXi license does not support API access. The vSphere servers must have If you have a cluster with DRS enabled, setting up [VM-VM Affinity Rules](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.resmgmt.doc/GUID-7297C302-378F-4AF2-9BD6-6EDB1E0A850A.html) is recommended. These rules allow VMs assigned the etcd and control-plane roles to operate on separate ESXi hosts when they are assigned to different node pools. This practice ensures that the failure of a single physical machine does not affect the availability of those planes. -## Creating a vSphere Cluster +## Creating a VMware vSphere Cluster 1. [Create your cloud credentials](#1-create-your-cloud-credentials) 2. [Create a node template with your cloud credentials](#2-create-a-node-template-with-your-cloud-credentials) diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md index 761df38fd1a..c2856714ccb 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md @@ -1,6 +1,6 @@ --- -title: Creating a vSphere Cluster -description: Use Rancher to create a vSphere cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. +title: Creating a VMware vSphere Cluster +description: Use Rancher to create a VMware vSphere cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. --- @@ -9,18 +9,18 @@ description: Use Rancher to create a vSphere cluster. It may consist of groups o import YouTube from '@site/src/components/YouTube' -By using Rancher with vSphere, you can bring cloud operations on-premises. +By using Rancher with VMware vSphere, you can bring cloud operations on-premises. Rancher can provision nodes in vSphere and install Kubernetes on them. When creating a Kubernetes cluster in vSphere, Rancher first provisions the specified number of virtual machines by communicating with the vCenter API. Then it installs Kubernetes on top of them. A vSphere cluster may consist of multiple groups of VMs with distinct properties, such as the amount of memory or the number of vCPUs. This grouping allows for fine-grained control over the sizing of nodes for each Kubernetes role. - [vSphere Enhancements in Rancher v2.3](#vsphere-enhancements-in-rancher-v23) -- [Creating a vSphere Cluster](#creating-a-vsphere-cluster) +- [Creating a vSphere Cluster](#creating-a-vmware-vsphere-cluster) - [Provisioning Storage](#provisioning-storage) - [Enabling the vSphere Cloud Provider](#enabling-the-vsphere-cloud-provider) -## vSphere Enhancements in Rancher v2.3 +## VMware vSphere Enhancements in Rancher v2.3 The vSphere node templates have been updated, allowing you to bring cloud operations on-premises with the following enhancements: @@ -34,7 +34,7 @@ One of the biggest advantages of provisioning vSphere nodes with Rancher is that Node templates for vSphere have been updated so that when you create a node template with your vSphere credentials, the template is automatically populated with the same options for provisioning VMs that you have access to in the vSphere console. -For the fields to be populated, your setup needs to fulfill the [prerequisites.](provision-kubernetes-clusters-in-vsphere.md#preparation-in-vsphere) +For the fields to be populated, your setup needs to fulfill the [prerequisites.](provision-kubernetes-clusters-in-vsphere.md#preparation-in-vmware-vsphere) ### More Supported Operating Systems @@ -46,7 +46,7 @@ In this YouTube video, we demonstrate how to set up a node template with the new -## Creating a vSphere Cluster +## Creating a VMware vSphere Cluster In [this section,](provision-kubernetes-clusters-in-vsphere.md) you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in vSphere. @@ -54,7 +54,7 @@ In [this section,](provision-kubernetes-clusters-in-vsphere.md) you'll learn how For an example of how to provision storage in vSphere using Rancher, refer to [this section.](../../../../../advanced-user-guides/manage-clusters/create-kubernetes-persistent-storage/provisioning-storage-examples/vsphere-storage.md) In order to dynamically provision storage in vSphere, the vSphere provider must be [enabled.](../../set-up-cloud-providers/vsphere/vsphere.md) -## Enabling the vSphere Cloud Provider +## Enabling the VMware vSphere Cloud Provider When a cloud provider is set up in Rancher, the Rancher server can automatically provision new infrastructure for the cluster, including new nodes or persistent storage devices. diff --git a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-windows-clusters/use-windows-clusters.md b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-windows-clusters/use-windows-clusters.md index 3f4fa8bd794..2588acbdf61 100644 --- a/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-windows-clusters/use-windows-clusters.md +++ b/versioned_docs/version-2.5/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-windows-clusters/use-windows-clusters.md @@ -80,7 +80,7 @@ If you are configuring DHCP options sets for an AWS virtual private cloud, note > Some Linux operating systems accept multiple domain names separated by spaces. However, other Linux operating systems and Windows treat the value as a single domain, which results in unexpected behavior. If your DHCP options set is associated with a VPC that has instances with multiple operating systems, specify only one domain name. -### Rancher on vSphere with ESXi 6.7u2 and above +### Rancher on VMware vSphere with ESXi 6.7u2 and above If you are using Rancher on VMware vSphere with ESXi 6.7u2 or later with Red Hat Enterprise Linux 8.3, CentOS 8.3, or SUSE Enterprise Linux 15 SP2 or later, it is necessary to disable the `vmxnet3` virtual network adapter hardware offloading feature. Failure to do so will result in all network connections between pods on different cluster nodes to fail with timeout errors. All connections from Windows pods to critical services running on Linux nodes, such as CoreDNS, will fail as well. It is also possible that external connections may fail. This issue is the result of Linux distributions enabling the hardware offloading feature in `vmxnet3` and a bug in the `vmxnet3` hardware offloading feature that results in the discarding of packets for guest overlay traffic. To address this issue, it is necessary disable the `vmxnet3` hardware offloading feature. This setting does not survive reboot, so it is necessary to disable on every boot. The recommended course of action is to create a systemd unit file at `/etc/systemd/system/disable_hw_offloading.service`, which disables the `vmxnet3` hardware offloading feature on boot. A sample systemd unit file which disables the `vmxnet3` hardware offloading feature is as follows. Note that `` must be customized to the host `vmxnet3` network interface, e.g., `ens192`: diff --git a/versioned_docs/version-2.5/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md b/versioned_docs/version-2.5/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md index 9d24fb3c5db..73242a90abb 100644 --- a/versioned_docs/version-2.5/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md +++ b/versioned_docs/version-2.5/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md @@ -1,5 +1,5 @@ --- -title: Best Practices for Rancher Managed vSphere Clusters +title: Best Practices for Rancher Managed VMware vSphere Clusters --- diff --git a/versioned_docs/version-2.5/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md b/versioned_docs/version-2.5/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md index ea4697f8e76..1e73af430af 100644 --- a/versioned_docs/version-2.5/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md +++ b/versioned_docs/version-2.5/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md @@ -18,6 +18,6 @@ Configuring sensible monitoring and alerting rules is vital for running any prod Running well-built containers can greatly impact the overall performance and security of your environment. Refer to this [guide](tips-to-set-up-containers.md) for tips. -### Best Practices for Rancher Managed vSphere Clusters +### Best Practices for Rancher Managed VMware vSphere Clusters This [guide](rancher-managed-clusters-in-vsphere.md) outlines a reference architecture for provisioning downstream Rancher clusters in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. diff --git a/versioned_docs/version-2.5/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md b/versioned_docs/version-2.5/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md index 87d4f348d3a..42ffa1ac9cf 100644 --- a/versioned_docs/version-2.5/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md +++ b/versioned_docs/version-2.5/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md @@ -1,12 +1,12 @@ --- -title: Installing Rancher in a vSphere Environment +title: Installing Rancher in a VMware vSphere Environment --- -This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. +This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a VMware vSphere environment. It also desctibes standard vSphere best practices as documented by VMware.
Solution Overview
diff --git a/versioned_docs/version-2.5/reference-guides/best-practices/rancher-server/rancher-server.md b/versioned_docs/version-2.5/reference-guides/best-practices/rancher-server/rancher-server.md index 8e24e145482..8babb9636fd 100644 --- a/versioned_docs/version-2.5/reference-guides/best-practices/rancher-server/rancher-server.md +++ b/versioned_docs/version-2.5/reference-guides/best-practices/rancher-server/rancher-server.md @@ -16,6 +16,6 @@ Refer to this [guide](tips-for-running-rancher.md) for our general advice for se This [guide](rancher-deployment-strategy.md) is designed to help you choose whether a regional deployment strategy or a hub-and-spoke deployment strategy is better for a Rancher server that manages downstream Kubernetes clusters. -### Installing Rancher in a vSphere Environment +### Installing Rancher in a VMware vSphere Environment This [guide](on-premises-rancher-in-vsphere.md) outlines a reference architecture for installing Rancher in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. \ No newline at end of file diff --git a/versioned_docs/version-2.5/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md b/versioned_docs/version-2.5/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md index 49afe925b82..415859e8456 100644 --- a/versioned_docs/version-2.5/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md +++ b/versioned_docs/version-2.5/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md @@ -1,5 +1,5 @@ --- -title: VSphere Node Template Configuration +title: VMware vSphere Node Template Configuration --- diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md index 7168b4ab493..90f326de2d1 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md @@ -1,12 +1,12 @@ --- -title: Setting Up an In-tree vSphere Cloud Provider +title: Setting Up an In-tree VMware vSphere Cloud Provider --- -To set up the in-tree vSphere cloud provider, follow these steps while creating the vSphere cluster in Rancher: +To set up the in-tree VMware vSphere cloud provider, follow these steps while creating the vSphere cluster in Rancher: 1. Set **Cloud Provider** option to `Custom` or `Custom (In-Tree)`. 1. Click on **Edit as YAML** diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md index ddaf11149cc..d8e17dffb8d 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md @@ -1,12 +1,12 @@ --- -title: Setting Up an Out-of-tree vSphere Cloud Provider +title: Setting Up an Out-of-tree VMware vSphere Cloud Provider --- -Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. +Kubernetes is moving away from maintaining cloud providers in-tree. VMware vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. This page covers how to install the Cloud Provider Interface (CPI) and Cloud Storage Interface (CSI) plugins after setting up a cluster. @@ -25,7 +25,7 @@ Using the vSphere out-of-tree cloud provider requires Linux nodes and is not sup The Cloud Provider Interface (CPI) should be installed first before installing the Cloud Storage Interface (CSI). -### 1. Create a vSphere cluster +### 1. Create a VMware vSphere cluster 1. Click **☰ > Cluster Management**. 1. On the **Clusters** page, click **Create**. diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md index 3993c8b2c6a..9db16230e0d 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md @@ -95,7 +95,7 @@ Some Linux operating systems accept multiple domain names separated by spaces. H ::: -### Rancher on vSphere with ESXi 6.7u2 and above +### Rancher on VMware vSphere with ESXi 6.7u2 and above If you are using Rancher on VMware vSphere with ESXi 6.7u2 or later with Red Hat Enterprise Linux 8.3, CentOS 8.3, or SUSE Enterprise Linux 15 SP2 or later, it is necessary to disable the `vmxnet3` virtual network adapter hardware offloading feature. Failure to do so will result in all network connections between pods on different cluster nodes to fail with timeout errors. All connections from Windows pods to critical services running on Linux nodes, such as CoreDNS, will fail as well. It is also possible that external connections may fail. This issue is the result of Linux distributions enabling the hardware offloading feature in `vmxnet3` and a bug in the `vmxnet3` hardware offloading feature that results in the discarding of packets for guest overlay traffic. To address this issue, it is necessary disable the `vmxnet3` hardware offloading feature. This setting does not survive reboot, so it is necessary to disable on every boot. The recommended course of action is to create a systemd unit file at `/etc/systemd/system/disable_hw_offloading.service`, which disables the `vmxnet3` hardware offloading feature on boot. A sample systemd unit file which disables the `vmxnet3` hardware offloading feature is as follows. Note that `` must be customized to the host `vmxnet3` network interface, e.g., `ens192`: diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md index cfdd8da1f19..55e23a9235a 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md @@ -1,5 +1,5 @@ --- -title: Creating a vSphere Virtual Machine Template +title: Creating a VMware vSphere Virtual Machine Template --- diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md index d228c27c1e7..20eb639ad31 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md @@ -1,12 +1,12 @@ --- -title: Creating Credentials in the vSphere Console +title: Creating Credentials in the VMware vSphere Console --- -This section describes how to create a vSphere username and password. You will need to provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. +This section describes how to create a VMware vSphere username and password. You must provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. The following table lists the permissions required for the vSphere user account: diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md index 9aec21efdf6..ae1bcf2fcb2 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md @@ -1,12 +1,12 @@ --- -title: Provisioning Kubernetes Clusters in vSphere +title: Provisioning Kubernetes Clusters in VMware vSphere --- -In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in vSphere. +In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in VMware vSphere. First, you will set up your vSphere cloud credentials in Rancher. Then you will use your cloud credentials to create a node template, which Rancher will use to provision nodes in vSphere. @@ -17,16 +17,16 @@ For details on configuring the vSphere node template, refer to the [vSphere node For details on configuring RKE Kubernetes clusters in Rancher, refer to the [cluster configuration reference.](../../../../../reference-guides/cluster-configuration/rancher-server-configuration/rke1-cluster-configuration.md#rke-cluster-config-file-reference) -- [Preparation in vSphere](#preparation-in-vsphere) -- [Creating a vSphere Cluster](#creating-a-vsphere-cluster) +- [Preparation in vSphere](#preparation-in-vmware-vsphere) +- [Creating a vSphere Cluster](#creating-a-vmware-vsphere-cluster) -## Preparation in vSphere +## Preparation in VMware vSphere This section describes the requirements for setting up vSphere so that Rancher can provision VMs and clusters. The node templates are documented and tested with the vSphere Web Services API version 6.5. -### Create Credentials in vSphere +### Create Credentials in VMware vSphere Before proceeding to create a cluster, you must ensure that you have a vSphere user with sufficient permissions. When you set up a node template, the template will need to use these vSphere credentials. @@ -42,7 +42,7 @@ It must be ensured that the hosts running the Rancher server are able to establi See [Node Networking Requirements](../../../kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md#networking-requirements) for a detailed list of port requirements applicable for creating nodes on an infrastructure provider. -### Valid ESXi License for vSphere API Access +### Valid ESXi License for VMware vSphere API Access The free ESXi license does not support API access. The vSphere servers must have a valid or evaluation ESXi license. @@ -50,7 +50,7 @@ The free ESXi license does not support API access. The vSphere servers must have If you have a cluster with DRS enabled, setting up [VM-VM Affinity Rules](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.resmgmt.doc/GUID-7297C302-378F-4AF2-9BD6-6EDB1E0A850A.html) is recommended. These rules allow VMs assigned the etcd and control-plane roles to operate on separate ESXi hosts when they are assigned to different node pools. This practice ensures that the failure of a single physical machine does not affect the availability of those planes. -## Creating a vSphere Cluster +## Creating a VMware vSphere Cluster 1. [Create your cloud credentials](#1-create-your-cloud-credentials) 2. [Create a node template with your cloud credentials](#2-create-a-node-template-with-your-cloud-credentials) diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md index e1d63847e59..ed7dbb7cd35 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md @@ -1,6 +1,6 @@ --- -title: Creating a vSphere Cluster -description: Use Rancher to create a vSphere cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. +title: Creating a VMware vSphere Cluster +description: Use Rancher to create a VMware vSphere cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. --- @@ -9,13 +9,13 @@ description: Use Rancher to create a vSphere cluster. It may consist of groups o import YouTube from '@site/src/components/YouTube' -By using Rancher with vSphere, you can bring cloud operations on-premises. +By using Rancher with VMware vSphere, you can bring cloud operations on-premises. Rancher can provision nodes in vSphere and install Kubernetes on them. When creating a Kubernetes cluster in vSphere, Rancher first provisions the specified number of virtual machines by communicating with the vCenter API. Then it installs Kubernetes on top of them. A vSphere cluster may consist of multiple groups of VMs with distinct properties, such as the amount of memory or the number of vCPUs. This grouping allows for fine-grained control over the sizing of nodes for each Kubernetes role. -## vSphere Enhancements in Rancher v2.3 +## VMware vSphere Enhancements in Rancher v2.3 The vSphere node templates have been updated, allowing you to bring cloud operations on-premises with the following enhancements: @@ -33,7 +33,7 @@ It is not recommended to enable node auto-replace on a node pool of master nodes Node templates for vSphere have been updated so that when you create a node template with your vSphere credentials, the template is automatically populated with the same options for provisioning VMs that you have access to in the vSphere console. -For the fields to be populated, your setup needs to fulfill the [prerequisites.](provision-kubernetes-clusters-in-vsphere.md#preparation-in-vsphere) +For the fields to be populated, your setup needs to fulfill the [prerequisites.](provision-kubernetes-clusters-in-vsphere.md#preparation-in-vmware-vsphere) ### More Supported Operating Systems @@ -45,7 +45,7 @@ In this YouTube video, we demonstrate how to set up a node template with the new -## Creating a vSphere Cluster +## Creating a VMware vSphere Cluster In [this section,](provision-kubernetes-clusters-in-vsphere.md) you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in vSphere. @@ -53,7 +53,7 @@ In [this section,](provision-kubernetes-clusters-in-vsphere.md) you'll learn how For an example of how to provision storage in vSphere using Rancher, refer to [this section.](../../../manage-clusters/provisioning-storage-examples/vsphere-storage.md) In order to dynamically provision storage in vSphere, the vSphere provider must be enabled. Refer to [in-tree vSphere config](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md) and [out-of-tree vSphere config](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md). -## Enabling the vSphere Cloud Provider +## Enabling the VMware vSphere Cloud Provider When a cloud provider is set up in Rancher, the Rancher server can automatically provision new infrastructure for the cluster, including new nodes or persistent storage devices. diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md index a3d842f5265..072ca040fe0 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -67,7 +67,7 @@ In [Rancher Launched Kubernetes clusters](../../launch-kubernetes-with-rancher/l ### hostPath Volumes Before you create a hostPath volume, you need to set up an [extra_bind](https://rancher.com/docs/rke/latest/en/config-options/services/services-extras/#extra-binds/) in your cluster configuration. This will mount the path as a volume in your kubelets, which can then be used for hostPath volumes in your workloads. -### Migrating vSphere Cloud Provider from In-tree to Out-of-tree +### Migrating VMware vSphere Cloud Provider from In-tree to Out-of-tree Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. diff --git a/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md b/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md index 0bede8b10ae..9d818629963 100644 --- a/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md +++ b/versioned_docs/version-2.6/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md @@ -1,12 +1,12 @@ --- -title: vSphere Storage +title: VMware vSphere Storage --- -To provide stateful workloads with vSphere storage, we recommend creating a vSphereVolume StorageClass. This practice dynamically provisions vSphere storage when workloads request volumes through a PersistentVolumeClaim. +To provide stateful workloads with VMware vSphere storage, we recommend creating a vSphereVolume StorageClass. This practice dynamically provisions vSphere storage when workloads request volumes through a PersistentVolumeClaim. In order to dynamically provision storage in vSphere, the vSphere provider must be enabled. See the following pages for more: [Out-of-tree vSphere](../../../new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md) and [in-tree vSphere](../../../new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md). @@ -35,7 +35,7 @@ The following steps can also be performed using the `kubectl` command line tool. 5. Optionally, specify additional properties for this storage class under **Parameters**. Refer to the [vSphere storage documentation](https://github.com/vmware-archive/vsphere-storage-for-kubernetes/blob/master/documentation/storageclass.md) for details. 5. Click **Create**. -### Creating a Workload with a vSphere Volume +### Creating a Workload with a VMware vSphere Volume 1. In the left navigation bar, click **Workload**. 1. Click **Create**. diff --git a/versioned_docs/version-2.6/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md b/versioned_docs/version-2.6/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md index 9d24fb3c5db..73242a90abb 100644 --- a/versioned_docs/version-2.6/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md +++ b/versioned_docs/version-2.6/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md @@ -1,5 +1,5 @@ --- -title: Best Practices for Rancher Managed vSphere Clusters +title: Best Practices for Rancher Managed VMware vSphere Clusters --- diff --git a/versioned_docs/version-2.6/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md b/versioned_docs/version-2.6/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md index ea4697f8e76..1e73af430af 100644 --- a/versioned_docs/version-2.6/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md +++ b/versioned_docs/version-2.6/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md @@ -18,6 +18,6 @@ Configuring sensible monitoring and alerting rules is vital for running any prod Running well-built containers can greatly impact the overall performance and security of your environment. Refer to this [guide](tips-to-set-up-containers.md) for tips. -### Best Practices for Rancher Managed vSphere Clusters +### Best Practices for Rancher Managed VMware vSphere Clusters This [guide](rancher-managed-clusters-in-vsphere.md) outlines a reference architecture for provisioning downstream Rancher clusters in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. diff --git a/versioned_docs/version-2.6/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md b/versioned_docs/version-2.6/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md index 6d35b2dc2d8..b5b412a1bef 100644 --- a/versioned_docs/version-2.6/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md +++ b/versioned_docs/version-2.6/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md @@ -1,12 +1,12 @@ --- -title: Installing Rancher in a vSphere Environment +title: Installing Rancher in a VMware vSphere Environment --- -This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. +This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a VMware vSphere environment. It also desctibes standard vSphere best practices as documented by VMware.
Solution Overview
diff --git a/versioned_docs/version-2.6/reference-guides/best-practices/rancher-server/rancher-server.md b/versioned_docs/version-2.6/reference-guides/best-practices/rancher-server/rancher-server.md index 8e24e145482..8babb9636fd 100644 --- a/versioned_docs/version-2.6/reference-guides/best-practices/rancher-server/rancher-server.md +++ b/versioned_docs/version-2.6/reference-guides/best-practices/rancher-server/rancher-server.md @@ -16,6 +16,6 @@ Refer to this [guide](tips-for-running-rancher.md) for our general advice for se This [guide](rancher-deployment-strategy.md) is designed to help you choose whether a regional deployment strategy or a hub-and-spoke deployment strategy is better for a Rancher server that manages downstream Kubernetes clusters. -### Installing Rancher in a vSphere Environment +### Installing Rancher in a VMware vSphere Environment This [guide](on-premises-rancher-in-vsphere.md) outlines a reference architecture for installing Rancher in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. \ No newline at end of file diff --git a/versioned_docs/version-2.6/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md b/versioned_docs/version-2.6/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md index baae6beac1a..6cf739475e6 100644 --- a/versioned_docs/version-2.6/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md +++ b/versioned_docs/version-2.6/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md @@ -1,5 +1,5 @@ --- -title: VSphere Node Template Configuration +title: VMware vSphere Node Template Configuration --- diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md index 568dd02a936..ba726357f43 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md @@ -1,5 +1,5 @@ --- -title: Migrating vSphere In-tree to Out-of-tree +title: Migrating VMware vSphere In-tree to Out-of-tree --- @@ -10,7 +10,7 @@ title: Migrating vSphere In-tree to Out-of-tree Rancher Prime provides access to Rancher v2.7.11, a version of Rancher v2.7.x which supports Kubernetes 1.27. If you use Rancher v2.7.11 and upgrade to Kubernetes 1.27, you must use an out-of-tree cloud provider. ::: -Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. +Kubernetes is moving away from maintaining cloud providers in-tree. VMware vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. This page covers how to migrate from the in-tree vSphere cloud provider to out-of-tree, and manage the existing VMs post migration. diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md index 7168b4ab493..90f326de2d1 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md @@ -1,12 +1,12 @@ --- -title: Setting Up an In-tree vSphere Cloud Provider +title: Setting Up an In-tree VMware vSphere Cloud Provider --- -To set up the in-tree vSphere cloud provider, follow these steps while creating the vSphere cluster in Rancher: +To set up the in-tree VMware vSphere cloud provider, follow these steps while creating the vSphere cluster in Rancher: 1. Set **Cloud Provider** option to `Custom` or `Custom (In-Tree)`. 1. Click on **Edit as YAML** diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md index 43d581ab2a3..6e866abc971 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md @@ -1,12 +1,12 @@ --- -title: Setting Up an Out-of-tree vSphere Cloud Provider +title: Setting Up an Out-of-tree VMware vSphere Cloud Provider --- -Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. +Kubernetes is moving away from maintaining cloud providers in-tree. VMware vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. This page covers how to install the Cloud Provider Interface (CPI) and Cloud Storage Interface (CSI) plugins after setting up a cluster. @@ -25,7 +25,7 @@ Using the vSphere out-of-tree cloud provider requires Linux nodes and is not sup The Cloud Provider Interface (CPI) should be installed first before installing the Cloud Storage Interface (CSI). -### 1. Create a vSphere cluster +### 1. Create a VMware vSphere cluster 1. Click **☰ > Cluster Management**. 1. On the **Clusters** page, click **Create**. diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md index 4c5f811c5e6..4c796540c2e 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md @@ -86,7 +86,7 @@ Some Linux operating systems accept multiple domain names separated by spaces. H ::: -### Rancher on vSphere with ESXi 6.7u2 and above +### Rancher on VMware vSphere with ESXi 6.7u2 and above If you are using Rancher on VMware vSphere with ESXi 6.7u2 or later with Red Hat Enterprise Linux 8.3, CentOS 8.3, or SUSE Enterprise Linux 15 SP2 or later, it is necessary to disable the `vmxnet3` virtual network adapter hardware offloading feature. Failure to do so will result in all network connections between pods on different cluster nodes to fail with timeout errors. All connections from Windows pods to critical services running on Linux nodes, such as CoreDNS, will fail as well. It is also possible that external connections may fail. This issue is the result of Linux distributions enabling the hardware offloading feature in `vmxnet3` and a bug in the `vmxnet3` hardware offloading feature that results in the discarding of packets for guest overlay traffic. To address this issue, it is necessary disable the `vmxnet3` hardware offloading feature. This setting does not survive reboot, so it is necessary to disable on every boot. The recommended course of action is to create a systemd unit file at `/etc/systemd/system/disable_hw_offloading.service`, which disables the `vmxnet3` hardware offloading feature on boot. A sample systemd unit file which disables the `vmxnet3` hardware offloading feature is as follows. Note that `` must be customized to the host `vmxnet3` network interface, e.g., `ens192`: diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md index cfdd8da1f19..55e23a9235a 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md @@ -1,5 +1,5 @@ --- -title: Creating a vSphere Virtual Machine Template +title: Creating a VMware vSphere Virtual Machine Template --- diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md index d228c27c1e7..20eb639ad31 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md @@ -1,12 +1,12 @@ --- -title: Creating Credentials in the vSphere Console +title: Creating Credentials in the VMware vSphere Console --- -This section describes how to create a vSphere username and password. You will need to provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. +This section describes how to create a VMware vSphere username and password. You must provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. The following table lists the permissions required for the vSphere user account: diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md index 9aec21efdf6..ae1bcf2fcb2 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md @@ -1,12 +1,12 @@ --- -title: Provisioning Kubernetes Clusters in vSphere +title: Provisioning Kubernetes Clusters in VMware vSphere --- -In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in vSphere. +In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in VMware vSphere. First, you will set up your vSphere cloud credentials in Rancher. Then you will use your cloud credentials to create a node template, which Rancher will use to provision nodes in vSphere. @@ -17,16 +17,16 @@ For details on configuring the vSphere node template, refer to the [vSphere node For details on configuring RKE Kubernetes clusters in Rancher, refer to the [cluster configuration reference.](../../../../../reference-guides/cluster-configuration/rancher-server-configuration/rke1-cluster-configuration.md#rke-cluster-config-file-reference) -- [Preparation in vSphere](#preparation-in-vsphere) -- [Creating a vSphere Cluster](#creating-a-vsphere-cluster) +- [Preparation in vSphere](#preparation-in-vmware-vsphere) +- [Creating a vSphere Cluster](#creating-a-vmware-vsphere-cluster) -## Preparation in vSphere +## Preparation in VMware vSphere This section describes the requirements for setting up vSphere so that Rancher can provision VMs and clusters. The node templates are documented and tested with the vSphere Web Services API version 6.5. -### Create Credentials in vSphere +### Create Credentials in VMware vSphere Before proceeding to create a cluster, you must ensure that you have a vSphere user with sufficient permissions. When you set up a node template, the template will need to use these vSphere credentials. @@ -42,7 +42,7 @@ It must be ensured that the hosts running the Rancher server are able to establi See [Node Networking Requirements](../../../kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md#networking-requirements) for a detailed list of port requirements applicable for creating nodes on an infrastructure provider. -### Valid ESXi License for vSphere API Access +### Valid ESXi License for VMware vSphere API Access The free ESXi license does not support API access. The vSphere servers must have a valid or evaluation ESXi license. @@ -50,7 +50,7 @@ The free ESXi license does not support API access. The vSphere servers must have If you have a cluster with DRS enabled, setting up [VM-VM Affinity Rules](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.resmgmt.doc/GUID-7297C302-378F-4AF2-9BD6-6EDB1E0A850A.html) is recommended. These rules allow VMs assigned the etcd and control-plane roles to operate on separate ESXi hosts when they are assigned to different node pools. This practice ensures that the failure of a single physical machine does not affect the availability of those planes. -## Creating a vSphere Cluster +## Creating a VMware vSphere Cluster 1. [Create your cloud credentials](#1-create-your-cloud-credentials) 2. [Create a node template with your cloud credentials](#2-create-a-node-template-with-your-cloud-credentials) diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md index e1d63847e59..ed7dbb7cd35 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md @@ -1,6 +1,6 @@ --- -title: Creating a vSphere Cluster -description: Use Rancher to create a vSphere cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. +title: Creating a VMware vSphere Cluster +description: Use Rancher to create a VMware vSphere cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. --- @@ -9,13 +9,13 @@ description: Use Rancher to create a vSphere cluster. It may consist of groups o import YouTube from '@site/src/components/YouTube' -By using Rancher with vSphere, you can bring cloud operations on-premises. +By using Rancher with VMware vSphere, you can bring cloud operations on-premises. Rancher can provision nodes in vSphere and install Kubernetes on them. When creating a Kubernetes cluster in vSphere, Rancher first provisions the specified number of virtual machines by communicating with the vCenter API. Then it installs Kubernetes on top of them. A vSphere cluster may consist of multiple groups of VMs with distinct properties, such as the amount of memory or the number of vCPUs. This grouping allows for fine-grained control over the sizing of nodes for each Kubernetes role. -## vSphere Enhancements in Rancher v2.3 +## VMware vSphere Enhancements in Rancher v2.3 The vSphere node templates have been updated, allowing you to bring cloud operations on-premises with the following enhancements: @@ -33,7 +33,7 @@ It is not recommended to enable node auto-replace on a node pool of master nodes Node templates for vSphere have been updated so that when you create a node template with your vSphere credentials, the template is automatically populated with the same options for provisioning VMs that you have access to in the vSphere console. -For the fields to be populated, your setup needs to fulfill the [prerequisites.](provision-kubernetes-clusters-in-vsphere.md#preparation-in-vsphere) +For the fields to be populated, your setup needs to fulfill the [prerequisites.](provision-kubernetes-clusters-in-vsphere.md#preparation-in-vmware-vsphere) ### More Supported Operating Systems @@ -45,7 +45,7 @@ In this YouTube video, we demonstrate how to set up a node template with the new -## Creating a vSphere Cluster +## Creating a VMware vSphere Cluster In [this section,](provision-kubernetes-clusters-in-vsphere.md) you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in vSphere. @@ -53,7 +53,7 @@ In [this section,](provision-kubernetes-clusters-in-vsphere.md) you'll learn how For an example of how to provision storage in vSphere using Rancher, refer to [this section.](../../../manage-clusters/provisioning-storage-examples/vsphere-storage.md) In order to dynamically provision storage in vSphere, the vSphere provider must be enabled. Refer to [in-tree vSphere config](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md) and [out-of-tree vSphere config](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md). -## Enabling the vSphere Cloud Provider +## Enabling the VMware vSphere Cloud Provider When a cloud provider is set up in Rancher, the Rancher server can automatically provision new infrastructure for the cluster, including new nodes or persistent storage devices. diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md index a3d842f5265..072ca040fe0 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -67,7 +67,7 @@ In [Rancher Launched Kubernetes clusters](../../launch-kubernetes-with-rancher/l ### hostPath Volumes Before you create a hostPath volume, you need to set up an [extra_bind](https://rancher.com/docs/rke/latest/en/config-options/services/services-extras/#extra-binds/) in your cluster configuration. This will mount the path as a volume in your kubelets, which can then be used for hostPath volumes in your workloads. -### Migrating vSphere Cloud Provider from In-tree to Out-of-tree +### Migrating VMware vSphere Cloud Provider from In-tree to Out-of-tree Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md index 0bede8b10ae..9d818629963 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md @@ -1,12 +1,12 @@ --- -title: vSphere Storage +title: VMware vSphere Storage --- -To provide stateful workloads with vSphere storage, we recommend creating a vSphereVolume StorageClass. This practice dynamically provisions vSphere storage when workloads request volumes through a PersistentVolumeClaim. +To provide stateful workloads with VMware vSphere storage, we recommend creating a vSphereVolume StorageClass. This practice dynamically provisions vSphere storage when workloads request volumes through a PersistentVolumeClaim. In order to dynamically provision storage in vSphere, the vSphere provider must be enabled. See the following pages for more: [Out-of-tree vSphere](../../../new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md) and [in-tree vSphere](../../../new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md). @@ -35,7 +35,7 @@ The following steps can also be performed using the `kubectl` command line tool. 5. Optionally, specify additional properties for this storage class under **Parameters**. Refer to the [vSphere storage documentation](https://github.com/vmware-archive/vsphere-storage-for-kubernetes/blob/master/documentation/storageclass.md) for details. 5. Click **Create**. -### Creating a Workload with a vSphere Volume +### Creating a Workload with a VMware vSphere Volume 1. In the left navigation bar, click **Workload**. 1. Click **Create**. diff --git a/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md b/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md index 9d24fb3c5db..73242a90abb 100644 --- a/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md +++ b/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md @@ -1,5 +1,5 @@ --- -title: Best Practices for Rancher Managed vSphere Clusters +title: Best Practices for Rancher Managed VMware vSphere Clusters --- diff --git a/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md b/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md index ea4697f8e76..1e73af430af 100644 --- a/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md +++ b/versioned_docs/version-2.7/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md @@ -18,6 +18,6 @@ Configuring sensible monitoring and alerting rules is vital for running any prod Running well-built containers can greatly impact the overall performance and security of your environment. Refer to this [guide](tips-to-set-up-containers.md) for tips. -### Best Practices for Rancher Managed vSphere Clusters +### Best Practices for Rancher Managed VMware vSphere Clusters This [guide](rancher-managed-clusters-in-vsphere.md) outlines a reference architecture for provisioning downstream Rancher clusters in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. diff --git a/versioned_docs/version-2.7/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md b/versioned_docs/version-2.7/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md index 6d35b2dc2d8..b5b412a1bef 100644 --- a/versioned_docs/version-2.7/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md +++ b/versioned_docs/version-2.7/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md @@ -1,12 +1,12 @@ --- -title: Installing Rancher in a vSphere Environment +title: Installing Rancher in a VMware vSphere Environment --- -This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. +This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a VMware vSphere environment. It also desctibes standard vSphere best practices as documented by VMware.
Solution Overview
diff --git a/versioned_docs/version-2.7/reference-guides/best-practices/rancher-server/rancher-server.md b/versioned_docs/version-2.7/reference-guides/best-practices/rancher-server/rancher-server.md index 8e24e145482..8babb9636fd 100644 --- a/versioned_docs/version-2.7/reference-guides/best-practices/rancher-server/rancher-server.md +++ b/versioned_docs/version-2.7/reference-guides/best-practices/rancher-server/rancher-server.md @@ -16,6 +16,6 @@ Refer to this [guide](tips-for-running-rancher.md) for our general advice for se This [guide](rancher-deployment-strategy.md) is designed to help you choose whether a regional deployment strategy or a hub-and-spoke deployment strategy is better for a Rancher server that manages downstream Kubernetes clusters. -### Installing Rancher in a vSphere Environment +### Installing Rancher in a VMware vSphere Environment This [guide](on-premises-rancher-in-vsphere.md) outlines a reference architecture for installing Rancher in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. \ No newline at end of file diff --git a/versioned_docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md b/versioned_docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md index ace79a39ca7..cfc2ed183b7 100644 --- a/versioned_docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md +++ b/versioned_docs/version-2.7/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md @@ -1,5 +1,5 @@ --- -title: VSphere Node Template Configuration +title: VMware vSphere Node Template Configuration --- diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md index c93da30901c..df230919675 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/migrate-to-an-out-of-tree-cloud-provider/migrate-to-out-of-tree-vsphere.md @@ -1,12 +1,12 @@ --- -title: Migrating vSphere In-tree to Out-of-tree +title: Migrating VMware vSphere In-tree to Out-of-tree --- -Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. +Kubernetes is moving away from maintaining cloud providers in-tree. VMware vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. This page covers how to migrate from the in-tree vSphere cloud provider to out-of-tree, and manage the existing VMs post migration. diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md index 7168b4ab493..90f326de2d1 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md @@ -1,12 +1,12 @@ --- -title: Setting Up an In-tree vSphere Cloud Provider +title: Setting Up an In-tree VMware vSphere Cloud Provider --- -To set up the in-tree vSphere cloud provider, follow these steps while creating the vSphere cluster in Rancher: +To set up the in-tree VMware vSphere cloud provider, follow these steps while creating the vSphere cluster in Rancher: 1. Set **Cloud Provider** option to `Custom` or `Custom (In-Tree)`. 1. Click on **Edit as YAML** diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md index 43d581ab2a3..6e866abc971 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md @@ -1,12 +1,12 @@ --- -title: Setting Up an Out-of-tree vSphere Cloud Provider +title: Setting Up an Out-of-tree VMware vSphere Cloud Provider --- -Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. +Kubernetes is moving away from maintaining cloud providers in-tree. VMware vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. This page covers how to install the Cloud Provider Interface (CPI) and Cloud Storage Interface (CSI) plugins after setting up a cluster. @@ -25,7 +25,7 @@ Using the vSphere out-of-tree cloud provider requires Linux nodes and is not sup The Cloud Provider Interface (CPI) should be installed first before installing the Cloud Storage Interface (CSI). -### 1. Create a vSphere cluster +### 1. Create a VMware vSphere cluster 1. Click **☰ > Cluster Management**. 1. On the **Clusters** page, click **Create**. diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md index 4c5f811c5e6..4c796540c2e 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/use-windows-clusters/use-windows-clusters.md @@ -86,7 +86,7 @@ Some Linux operating systems accept multiple domain names separated by spaces. H ::: -### Rancher on vSphere with ESXi 6.7u2 and above +### Rancher on VMware vSphere with ESXi 6.7u2 and above If you are using Rancher on VMware vSphere with ESXi 6.7u2 or later with Red Hat Enterprise Linux 8.3, CentOS 8.3, or SUSE Enterprise Linux 15 SP2 or later, it is necessary to disable the `vmxnet3` virtual network adapter hardware offloading feature. Failure to do so will result in all network connections between pods on different cluster nodes to fail with timeout errors. All connections from Windows pods to critical services running on Linux nodes, such as CoreDNS, will fail as well. It is also possible that external connections may fail. This issue is the result of Linux distributions enabling the hardware offloading feature in `vmxnet3` and a bug in the `vmxnet3` hardware offloading feature that results in the discarding of packets for guest overlay traffic. To address this issue, it is necessary disable the `vmxnet3` hardware offloading feature. This setting does not survive reboot, so it is necessary to disable on every boot. The recommended course of action is to create a systemd unit file at `/etc/systemd/system/disable_hw_offloading.service`, which disables the `vmxnet3` hardware offloading feature on boot. A sample systemd unit file which disables the `vmxnet3` hardware offloading feature is as follows. Note that `` must be customized to the host `vmxnet3` network interface, e.g., `ens192`: diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md index cfdd8da1f19..55e23a9235a 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-a-vm-template.md @@ -1,5 +1,5 @@ --- -title: Creating a vSphere Virtual Machine Template +title: Creating a VMware vSphere Virtual Machine Template --- diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md index d228c27c1e7..20eb639ad31 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/create-credentials.md @@ -1,12 +1,12 @@ --- -title: Creating Credentials in the vSphere Console +title: Creating Credentials in the VMware vSphere Console --- -This section describes how to create a vSphere username and password. You will need to provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. +This section describes how to create a VMware vSphere username and password. You must provide these vSphere credentials to Rancher, which allows Rancher to provision resources in vSphere. The following table lists the permissions required for the vSphere user account: diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md index 92eb0a4f6f9..d7835c437a8 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/provision-kubernetes-clusters-in-vsphere.md @@ -1,12 +1,12 @@ --- -title: Provisioning Kubernetes Clusters in vSphere +title: Provisioning Kubernetes Clusters in VMware vSphere --- -In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in vSphere. +In this section, you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in VMware vSphere. First, you will set up your vSphere cloud credentials in Rancher. Then you will use your cloud credentials to create a node template, which Rancher will use to provision nodes in vSphere. @@ -17,16 +17,16 @@ For details on configuring the vSphere node template, refer to the [vSphere node For details on configuring RKE Kubernetes clusters in Rancher, refer to the [cluster configuration reference.](../../../../../reference-guides/cluster-configuration/rancher-server-configuration/rke1-cluster-configuration.md#rke-cluster-config-file-reference) -- [Preparation in vSphere](#preparation-in-vsphere) -- [Creating a vSphere Cluster](#creating-a-vsphere-cluster) +- [Preparation in vSphere](#preparation-in-vmware-vsphere) +- [Creating a vSphere Cluster](#creating-a-vmware-vsphere-cluster) -## Preparation in vSphere +## Preparation in VMware vSphere This section describes the requirements for setting up vSphere so that Rancher can provision VMs and clusters. The node templates are documented and tested with the vSphere Web Services API version 6.5. -### Create Credentials in vSphere +### Create Credentials in VMware vSphere Before proceeding to create a cluster, you must ensure that you have a vSphere user with sufficient permissions. When you set up a node template, the template will need to use these vSphere credentials. @@ -42,7 +42,7 @@ It must be ensured that the hosts running the Rancher server are able to establi See [Node Networking Requirements](../../../kubernetes-clusters-in-rancher-setup/node-requirements-for-rancher-managed-clusters.md#networking-requirements) for a detailed list of port requirements applicable for creating nodes on an infrastructure provider. -### Valid ESXi License for vSphere API Access +### Valid ESXi License for VMware vSphere API Access The free ESXi license does not support API access. The vSphere servers must have a valid or evaluation ESXi license. @@ -50,7 +50,7 @@ The free ESXi license does not support API access. The vSphere servers must have If you have a cluster with DRS enabled, setting up [VM-VM Affinity Rules](https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.resmgmt.doc/GUID-7297C302-378F-4AF2-9BD6-6EDB1E0A850A.html) is recommended. These rules allow VMs assigned the etcd and control-plane roles to operate on separate ESXi hosts when they are assigned to different node pools. This practice ensures that the failure of a single physical machine does not affect the availability of those planes. -## Creating a vSphere Cluster +## Creating a VMware vSphere Cluster 1. [Create your cloud credentials](#1-create-your-cloud-credentials) 2. [Create a node template with your cloud credentials](#2-create-a-node-template-with-your-cloud-credentials) diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md index e1d63847e59..ed7dbb7cd35 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere.md @@ -1,6 +1,6 @@ --- -title: Creating a vSphere Cluster -description: Use Rancher to create a vSphere cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. +title: Creating a VMware vSphere Cluster +description: Use Rancher to create a VMware vSphere cluster. It may consist of groups of VMs with distinct properties which allow for fine-grained control over the sizing of nodes. --- @@ -9,13 +9,13 @@ description: Use Rancher to create a vSphere cluster. It may consist of groups o import YouTube from '@site/src/components/YouTube' -By using Rancher with vSphere, you can bring cloud operations on-premises. +By using Rancher with VMware vSphere, you can bring cloud operations on-premises. Rancher can provision nodes in vSphere and install Kubernetes on them. When creating a Kubernetes cluster in vSphere, Rancher first provisions the specified number of virtual machines by communicating with the vCenter API. Then it installs Kubernetes on top of them. A vSphere cluster may consist of multiple groups of VMs with distinct properties, such as the amount of memory or the number of vCPUs. This grouping allows for fine-grained control over the sizing of nodes for each Kubernetes role. -## vSphere Enhancements in Rancher v2.3 +## VMware vSphere Enhancements in Rancher v2.3 The vSphere node templates have been updated, allowing you to bring cloud operations on-premises with the following enhancements: @@ -33,7 +33,7 @@ It is not recommended to enable node auto-replace on a node pool of master nodes Node templates for vSphere have been updated so that when you create a node template with your vSphere credentials, the template is automatically populated with the same options for provisioning VMs that you have access to in the vSphere console. -For the fields to be populated, your setup needs to fulfill the [prerequisites.](provision-kubernetes-clusters-in-vsphere.md#preparation-in-vsphere) +For the fields to be populated, your setup needs to fulfill the [prerequisites.](provision-kubernetes-clusters-in-vsphere.md#preparation-in-vmware-vsphere) ### More Supported Operating Systems @@ -45,7 +45,7 @@ In this YouTube video, we demonstrate how to set up a node template with the new -## Creating a vSphere Cluster +## Creating a VMware vSphere Cluster In [this section,](provision-kubernetes-clusters-in-vsphere.md) you'll learn how to use Rancher to install an [RKE](https://rancher.com/docs/rke/latest/en/) Kubernetes cluster in vSphere. @@ -53,7 +53,7 @@ In [this section,](provision-kubernetes-clusters-in-vsphere.md) you'll learn how For an example of how to provision storage in vSphere using Rancher, refer to [this section.](../../../manage-clusters/provisioning-storage-examples/vsphere-storage.md) In order to dynamically provision storage in vSphere, the vSphere provider must be enabled. Refer to [in-tree vSphere config](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md) and [out-of-tree vSphere config](../../../kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md). -## Enabling the vSphere Cloud Provider +## Enabling the VMware vSphere Cloud Provider When a cloud provider is set up in Rancher, the Rancher server can automatically provision new infrastructure for the cluster, including new nodes or persistent storage devices. diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md index 7ada9cd85af..67a8c9dffbd 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/create-kubernetes-persistent-storage/create-kubernetes-persistent-storage.md @@ -67,7 +67,7 @@ In [Rancher Launched Kubernetes clusters](../../launch-kubernetes-with-rancher/l ### hostPath Volumes Before you create a hostPath volume, you need to set up an [extra_bind](https://rancher.com/docs/rke/latest/en/config-options/services/services-extras/#extra-binds/) in your cluster configuration. This will mount the path as a volume in your kubelets, which can then be used for hostPath volumes in your workloads. -### Migrating vSphere Cloud Provider from In-tree to Out-of-tree +### Migrating VMware vSphere Cloud Provider from In-tree to Out-of-tree Kubernetes is moving away from maintaining cloud providers in-tree. vSphere has an out-of-tree cloud provider that can be used by installing the vSphere cloud provider and cloud storage plugins. diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md index 0bede8b10ae..9d818629963 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/manage-clusters/provisioning-storage-examples/vsphere-storage.md @@ -1,12 +1,12 @@ --- -title: vSphere Storage +title: VMware vSphere Storage --- -To provide stateful workloads with vSphere storage, we recommend creating a vSphereVolume StorageClass. This practice dynamically provisions vSphere storage when workloads request volumes through a PersistentVolumeClaim. +To provide stateful workloads with VMware vSphere storage, we recommend creating a vSphereVolume StorageClass. This practice dynamically provisions vSphere storage when workloads request volumes through a PersistentVolumeClaim. In order to dynamically provision storage in vSphere, the vSphere provider must be enabled. See the following pages for more: [Out-of-tree vSphere](../../../new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-out-of-tree-vsphere.md) and [in-tree vSphere](../../../new-user-guides/kubernetes-clusters-in-rancher-setup/set-up-cloud-providers/configure-in-tree-vsphere.md). @@ -35,7 +35,7 @@ The following steps can also be performed using the `kubectl` command line tool. 5. Optionally, specify additional properties for this storage class under **Parameters**. Refer to the [vSphere storage documentation](https://github.com/vmware-archive/vsphere-storage-for-kubernetes/blob/master/documentation/storageclass.md) for details. 5. Click **Create**. -### Creating a Workload with a vSphere Volume +### Creating a Workload with a VMware vSphere Volume 1. In the left navigation bar, click **Workload**. 1. Click **Create**. diff --git a/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md b/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md index 9d24fb3c5db..73242a90abb 100644 --- a/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md +++ b/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters-in-vsphere.md @@ -1,5 +1,5 @@ --- -title: Best Practices for Rancher Managed vSphere Clusters +title: Best Practices for Rancher Managed VMware vSphere Clusters --- diff --git a/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md b/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md index ea4697f8e76..1e73af430af 100644 --- a/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md +++ b/versioned_docs/version-2.8/reference-guides/best-practices/rancher-managed-clusters/rancher-managed-clusters.md @@ -18,6 +18,6 @@ Configuring sensible monitoring and alerting rules is vital for running any prod Running well-built containers can greatly impact the overall performance and security of your environment. Refer to this [guide](tips-to-set-up-containers.md) for tips. -### Best Practices for Rancher Managed vSphere Clusters +### Best Practices for Rancher Managed VMware vSphere Clusters This [guide](rancher-managed-clusters-in-vsphere.md) outlines a reference architecture for provisioning downstream Rancher clusters in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. diff --git a/versioned_docs/version-2.8/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md b/versioned_docs/version-2.8/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md index 6d35b2dc2d8..b5b412a1bef 100644 --- a/versioned_docs/version-2.8/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md +++ b/versioned_docs/version-2.8/reference-guides/best-practices/rancher-server/on-premises-rancher-in-vsphere.md @@ -1,12 +1,12 @@ --- -title: Installing Rancher in a vSphere Environment +title: Installing Rancher in a VMware vSphere Environment --- -This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. +This guide outlines a reference architecture for installing Rancher on an RKE Kubernetes cluster in a VMware vSphere environment. It also desctibes standard vSphere best practices as documented by VMware.
Solution Overview
diff --git a/versioned_docs/version-2.8/reference-guides/best-practices/rancher-server/rancher-server.md b/versioned_docs/version-2.8/reference-guides/best-practices/rancher-server/rancher-server.md index 8e24e145482..8babb9636fd 100644 --- a/versioned_docs/version-2.8/reference-guides/best-practices/rancher-server/rancher-server.md +++ b/versioned_docs/version-2.8/reference-guides/best-practices/rancher-server/rancher-server.md @@ -16,6 +16,6 @@ Refer to this [guide](tips-for-running-rancher.md) for our general advice for se This [guide](rancher-deployment-strategy.md) is designed to help you choose whether a regional deployment strategy or a hub-and-spoke deployment strategy is better for a Rancher server that manages downstream Kubernetes clusters. -### Installing Rancher in a vSphere Environment +### Installing Rancher in a VMware vSphere Environment This [guide](on-premises-rancher-in-vsphere.md) outlines a reference architecture for installing Rancher in a vSphere environment, in addition to standard vSphere best practices as documented by VMware. \ No newline at end of file diff --git a/versioned_docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md b/versioned_docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md index 135c3edf7cc..8c986468f9f 100644 --- a/versioned_docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md +++ b/versioned_docs/version-2.8/reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere.md @@ -1,5 +1,5 @@ --- -title: VSphere Node Template Configuration +title: VMware vSphere Node Template Configuration --- diff --git a/versioned_sidebars/version-2.0-2.4-sidebars.json b/versioned_sidebars/version-2.0-2.4-sidebars.json index c3ad1072c3a..b3ad3ec4483 100644 --- a/versioned_sidebars/version-2.0-2.4-sidebars.json +++ b/versioned_sidebars/version-2.0-2.4-sidebars.json @@ -459,7 +459,7 @@ "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster", { "type": "category", - "label": "Creating a vSphere Cluster", + "label": "Creating a VMware vSphere Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere" @@ -977,7 +977,7 @@ "reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/azure", { "type": "category", - "label": "Creating a vSphere Cluster", + "label": "Creating a VMware vSphere Cluster", "link": { "type": "doc", "id": "reference-guides/cluster-configuration/downstream-cluster-configuration/node-template-configuration/vsphere/vsphere" diff --git a/versioned_sidebars/version-2.5-sidebars.json b/versioned_sidebars/version-2.5-sidebars.json index b9ac6a84bce..0c8e7f7958f 100644 --- a/versioned_sidebars/version-2.5-sidebars.json +++ b/versioned_sidebars/version-2.5-sidebars.json @@ -320,7 +320,7 @@ "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster", { "type": "category", - "label": "Creating a vSphere Cluster", + "label": "Creating a VMware vSphere Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere" @@ -368,7 +368,7 @@ }, { "type": "category", - "label": "Creating a vSphere Cluster", + "label": "Creating a VMware vSphere Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/kubernetes-clusters-in-rancher-setup/launch-kubernetes-with-rancher/set-up-cloud-providers/vsphere/vsphere" diff --git a/versioned_sidebars/version-2.6-sidebars.json b/versioned_sidebars/version-2.6-sidebars.json index 8e52facf872..2c7495a1c84 100644 --- a/versioned_sidebars/version-2.6-sidebars.json +++ b/versioned_sidebars/version-2.6-sidebars.json @@ -492,7 +492,7 @@ "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster", { "type": "category", - "label": "Creating a vSphere Cluster", + "label": "Creating a VMware vSphere Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere" diff --git a/versioned_sidebars/version-2.7-sidebars.json b/versioned_sidebars/version-2.7-sidebars.json index 11fc615225d..02406ce9072 100644 --- a/versioned_sidebars/version-2.7-sidebars.json +++ b/versioned_sidebars/version-2.7-sidebars.json @@ -503,7 +503,7 @@ "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster", { "type": "category", - "label": "Creating a vSphere Cluster", + "label": "Creating a VMware vSphere Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere" diff --git a/versioned_sidebars/version-2.8-sidebars.json b/versioned_sidebars/version-2.8-sidebars.json index de5f50aa6ec..09ba1a87f49 100644 --- a/versioned_sidebars/version-2.8-sidebars.json +++ b/versioned_sidebars/version-2.8-sidebars.json @@ -503,7 +503,7 @@ "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/create-an-azure-cluster", { "type": "category", - "label": "Creating a vSphere Cluster", + "label": "Creating a VMware vSphere Cluster", "link": { "type": "doc", "id": "how-to-guides/new-user-guides/launch-kubernetes-with-rancher/use-new-nodes-in-an-infra-provider/vsphere/vsphere" From 3c0b963f9f2c4830327048f21359173a9ba32647 Mon Sep 17 00:00:00 2001 From: Marty Hernandez Avedon Date: Mon, 6 May 2024 16:54:51 -0400 Subject: [PATCH 66/67] #1182 add warning note for node driver deletion on vmware (#1242) * 1182 Add warning note for node driver deletion on vmware * fix typos * fix headings, reword for clarity * typos, formating, added bit about providers & instructions to view * reword * added back steps about cluster management page * Apply suggestions from code review Co-authored-by: Billy Tat * versioning --------- Co-authored-by: Billy Tat --- .../manage-node-drivers.md | 32 +++++++++++++------ .../manage-node-drivers.md | 32 +++++++++++++------ .../manage-node-drivers.md | 32 +++++++++++++------ 3 files changed, 69 insertions(+), 27 deletions(-) diff --git a/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md b/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md index 12c901bfd41..a1539a663a7 100644 --- a/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md +++ b/docs/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md @@ -6,11 +6,11 @@ title: Node Drivers -Node drivers are used to provision hosts, which Rancher uses to launch and manage Kubernetes clusters. A node driver is the same as a [Docker Machine driver](https://docs.docker.com/machine/drivers/). The availability of which node driver to display when creating node templates is defined based on the node driver's status. Only `active` node drivers will be displayed as an option for creating node templates. By default, Rancher is packaged with many existing Docker Machine drivers, but you can also create custom node drivers to add to Rancher. +A node driver is the same as a [Docker Machine driver](https://docs.docker.com/machine/drivers/). Node drivers are used to provision hosts, which Rancher uses to launch and manage Kubernetes clusters. By default, Rancher is packaged with many node drivers, but you can also create and add custom node drivers to Rancher. -If there are specific node drivers that you don't want to show to your users, you would need to de-activate these node drivers. +Only `Active` node drivers are displayed in the Rancher UI when you create node templates. If there are specific node drivers that you don't want to show your users, you must deactivate these node drivers. -#### Managing Node Drivers +## Managing Node Drivers :::note Prerequisites: @@ -21,17 +21,31 @@ To create, edit, or delete drivers, you need _one_ of the following permissions: ::: -## Activating/Deactivating Node Drivers +### Activating/Deactivating Node Drivers -By default, Rancher only activates drivers for the most popular cloud providers, Amazon EC2, Azure, DigitalOcean, Linode and vSphere. If you want to show or hide any node driver, you can change its status. +By default, Rancher only activates drivers for the most popular cloud providers, such as Amazon EC2, Azure, DigitalOcean, Linode and vSphere. If you want to show or hide any node driver, you can change its status. 1. In the upper left corner, click **☰ > Cluster Management**. +1. In the left navigation menu, click **Drivers**. +1. On the **Node Drivers** tab, select the driver that you wish to activate or deactivate and click **⋮ > Activate** or **⋮ > Deactivate**. -2. In the left navigation menu, click **Drivers**. +:::danger -2. On the **Node Drivers** tab, select the driver that you wish to activate or deactivate and click **⋮ > Activate** or **⋮ > Deactivate**. +You can lose access to clusters after deactivating a node driver. -## Adding Custom Node Drivers +Deactivating a node driver doesn't just affect its visibility in the Rancher UI. When you deactivate or delete a node driver, any nodes deployed with that driver become inaccessible. + +For example, if you deactivate a vSphere node driver to hide it in the UI, and you have a vSphere cluster that was deployed with that driver, the initial node in the cluster will fail, and the entire cluster will become inaccessible. Attempts to delete the vSphere nodes will fail, with nodes stuck in an extended `Removing` state. + +Before you deactivate a node driver, make sure that it has no associated clusters. One way to check is to see if the respective platform for a driver is listed among your clusters: + +1. In the upper left corner, click **☰ > Cluster Management**. +1. Select **Clusters**. +1. Check the **Provider** column of the table for instances of the node driver you are deactivating. + +::: + +### Adding Custom Node Drivers If you want to use a node driver that Rancher doesn't support out-of-the-box, you can add that provider's driver in order to start using them to create node templates and eventually node pools for your Kubernetes cluster. @@ -40,6 +54,6 @@ If you want to use a node driver that Rancher doesn't support out-of-the-box, yo 1. On **Node Drivers** tab, click **Add Node Driver**. 1. Complete the **Add Node Driver** form. Then click **Create**. -### Developing your own node driver +### Developing Your Own Node Drivers Node drivers are implemented with [Docker Machine](https://docs.docker.com/machine/). diff --git a/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md b/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md index 44256b18da0..a1539a663a7 100644 --- a/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md +++ b/versioned_docs/version-2.7/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md @@ -6,11 +6,11 @@ title: Node Drivers -Node drivers are used to provision hosts, which Rancher uses to launch and manage Kubernetes clusters. A node driver is the same as a [Docker Machine driver](https://docs.docker.com/machine/drivers/). The availability of which node driver to display when creating node templates is defined based on the node driver's status. Only `active` node drivers will be displayed as an option for creating node templates. By default, Rancher is packaged with many existing Docker Machine drivers, but you can also create custom node drivers to add to Rancher. +A node driver is the same as a [Docker Machine driver](https://docs.docker.com/machine/drivers/). Node drivers are used to provision hosts, which Rancher uses to launch and manage Kubernetes clusters. By default, Rancher is packaged with many node drivers, but you can also create and add custom node drivers to Rancher. -If there are specific node drivers that you don't want to show to your users, you would need to de-activate these node drivers. +Only `Active` node drivers are displayed in the Rancher UI when you create node templates. If there are specific node drivers that you don't want to show your users, you must deactivate these node drivers. -#### Managing Node Drivers +## Managing Node Drivers :::note Prerequisites: @@ -21,17 +21,31 @@ To create, edit, or delete drivers, you need _one_ of the following permissions: ::: -## Activating/Deactivating Node Drivers +### Activating/Deactivating Node Drivers -By default, Rancher only activates drivers for the most popular cloud providers, Amazon EC2, Azure, DigitalOcean and vSphere. If you want to show or hide any node driver, you can change its status. +By default, Rancher only activates drivers for the most popular cloud providers, such as Amazon EC2, Azure, DigitalOcean, Linode and vSphere. If you want to show or hide any node driver, you can change its status. 1. In the upper left corner, click **☰ > Cluster Management**. +1. In the left navigation menu, click **Drivers**. +1. On the **Node Drivers** tab, select the driver that you wish to activate or deactivate and click **⋮ > Activate** or **⋮ > Deactivate**. -2. In the left navigation menu, click **Drivers**. +:::danger -2. On the **Node Drivers** tab, select the driver that you wish to activate or deactivate and click **⋮ > Activate** or **⋮ > Deactivate**. +You can lose access to clusters after deactivating a node driver. -## Adding Custom Node Drivers +Deactivating a node driver doesn't just affect its visibility in the Rancher UI. When you deactivate or delete a node driver, any nodes deployed with that driver become inaccessible. + +For example, if you deactivate a vSphere node driver to hide it in the UI, and you have a vSphere cluster that was deployed with that driver, the initial node in the cluster will fail, and the entire cluster will become inaccessible. Attempts to delete the vSphere nodes will fail, with nodes stuck in an extended `Removing` state. + +Before you deactivate a node driver, make sure that it has no associated clusters. One way to check is to see if the respective platform for a driver is listed among your clusters: + +1. In the upper left corner, click **☰ > Cluster Management**. +1. Select **Clusters**. +1. Check the **Provider** column of the table for instances of the node driver you are deactivating. + +::: + +### Adding Custom Node Drivers If you want to use a node driver that Rancher doesn't support out-of-the-box, you can add that provider's driver in order to start using them to create node templates and eventually node pools for your Kubernetes cluster. @@ -40,6 +54,6 @@ If you want to use a node driver that Rancher doesn't support out-of-the-box, yo 1. On **Node Drivers** tab, click **Add Node Driver**. 1. Complete the **Add Node Driver** form. Then click **Create**. -### Developing your own node driver +### Developing Your Own Node Drivers Node drivers are implemented with [Docker Machine](https://docs.docker.com/machine/). diff --git a/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md b/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md index 44256b18da0..a1539a663a7 100644 --- a/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md +++ b/versioned_docs/version-2.8/how-to-guides/new-user-guides/authentication-permissions-and-global-configuration/about-provisioning-drivers/manage-node-drivers.md @@ -6,11 +6,11 @@ title: Node Drivers -Node drivers are used to provision hosts, which Rancher uses to launch and manage Kubernetes clusters. A node driver is the same as a [Docker Machine driver](https://docs.docker.com/machine/drivers/). The availability of which node driver to display when creating node templates is defined based on the node driver's status. Only `active` node drivers will be displayed as an option for creating node templates. By default, Rancher is packaged with many existing Docker Machine drivers, but you can also create custom node drivers to add to Rancher. +A node driver is the same as a [Docker Machine driver](https://docs.docker.com/machine/drivers/). Node drivers are used to provision hosts, which Rancher uses to launch and manage Kubernetes clusters. By default, Rancher is packaged with many node drivers, but you can also create and add custom node drivers to Rancher. -If there are specific node drivers that you don't want to show to your users, you would need to de-activate these node drivers. +Only `Active` node drivers are displayed in the Rancher UI when you create node templates. If there are specific node drivers that you don't want to show your users, you must deactivate these node drivers. -#### Managing Node Drivers +## Managing Node Drivers :::note Prerequisites: @@ -21,17 +21,31 @@ To create, edit, or delete drivers, you need _one_ of the following permissions: ::: -## Activating/Deactivating Node Drivers +### Activating/Deactivating Node Drivers -By default, Rancher only activates drivers for the most popular cloud providers, Amazon EC2, Azure, DigitalOcean and vSphere. If you want to show or hide any node driver, you can change its status. +By default, Rancher only activates drivers for the most popular cloud providers, such as Amazon EC2, Azure, DigitalOcean, Linode and vSphere. If you want to show or hide any node driver, you can change its status. 1. In the upper left corner, click **☰ > Cluster Management**. +1. In the left navigation menu, click **Drivers**. +1. On the **Node Drivers** tab, select the driver that you wish to activate or deactivate and click **⋮ > Activate** or **⋮ > Deactivate**. -2. In the left navigation menu, click **Drivers**. +:::danger -2. On the **Node Drivers** tab, select the driver that you wish to activate or deactivate and click **⋮ > Activate** or **⋮ > Deactivate**. +You can lose access to clusters after deactivating a node driver. -## Adding Custom Node Drivers +Deactivating a node driver doesn't just affect its visibility in the Rancher UI. When you deactivate or delete a node driver, any nodes deployed with that driver become inaccessible. + +For example, if you deactivate a vSphere node driver to hide it in the UI, and you have a vSphere cluster that was deployed with that driver, the initial node in the cluster will fail, and the entire cluster will become inaccessible. Attempts to delete the vSphere nodes will fail, with nodes stuck in an extended `Removing` state. + +Before you deactivate a node driver, make sure that it has no associated clusters. One way to check is to see if the respective platform for a driver is listed among your clusters: + +1. In the upper left corner, click **☰ > Cluster Management**. +1. Select **Clusters**. +1. Check the **Provider** column of the table for instances of the node driver you are deactivating. + +::: + +### Adding Custom Node Drivers If you want to use a node driver that Rancher doesn't support out-of-the-box, you can add that provider's driver in order to start using them to create node templates and eventually node pools for your Kubernetes cluster. @@ -40,6 +54,6 @@ If you want to use a node driver that Rancher doesn't support out-of-the-box, yo 1. On **Node Drivers** tab, click **Add Node Driver**. 1. Complete the **Add Node Driver** form. Then click **Create**. -### Developing your own node driver +### Developing Your Own Node Drivers Node drivers are implemented with [Docker Machine](https://docs.docker.com/machine/). From 7d3d40ae8368a33783958648881dbbf5047ca16b Mon Sep 17 00:00:00 2001 From: Carlos Salas Date: Tue, 7 May 2024 16:59:19 +0200 Subject: [PATCH 67/67] docs: edit hosted providers upstream/config sync (#1258) * docs: edit hosted providers specification sync Signed-off-by: Carlos Salas * docs: 2.7 edit hosted providers specification sync Signed-off-by: Carlos Salas * docs: 2.8 edit hosted providers specification sync Signed-off-by: Carlos Salas * Apply suggestions from code review --------- Signed-off-by: Carlos Salas Co-authored-by: Marty Hernandez Avedon --- .../rancher-server-configuration/sync-clusters.md | 4 ++++ .../rancher-server-configuration/sync-clusters.md | 4 ++++ .../rancher-server-configuration/sync-clusters.md | 4 ++++ 3 files changed, 12 insertions(+) diff --git a/docs/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md b/docs/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md index 53e7a4c6d9c..8ee65f0c248 100644 --- a/docs/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md +++ b/docs/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md @@ -40,6 +40,10 @@ The AKSConfig, EKSConfig or GKEConfig represents the desired state. Nil values a UpstreamSpec represents the cluster as it is in the hosted Kubernetes provider. It's refreshed every 5 minutes. After the UpstreamSpec is refreshed, Rancher checks if the cluster has an update in progress. If it's currently updating, nothing further is done. If it is not currently updating, any managed fields on AKSConfig, EKSConfig or GKEConfig are overwritten with their corresponding value from the recently updated UpstreamSpec. +:::warning +When you import a cluster from a cloud provider into Rancher, UpstreamSpec represents the cluster state and Config is empty. If you then update the imported cluster through the Rancher UI, both UpstreamSpec and Config become non-null. Any further updates to the cluster should be applied through Rancher. This is because there is no safe way to determine if changes originating from UpstreamSpec represent the desired state or just a mismatch with Config. If you update the imported cluster through the cloud provider console after you apply any updates through the Rancher UI, the controller will deploy a rollback and the content of Config will be considered the desired state. +::: + The effective desired state can be thought of as the UpstreamSpec, plus all non-nil fields in the AKSConfig, EKSConfig or GKEConfig. This is what is displayed in the UI. If Rancher and another source attempt to update a cluster at the same time, or within 5 minutes of an update finishing, any managed fields are likely to get caught in a race condition. To use EKS as an example, a cluster may have PrivateAccess as a managed field. If PrivateAccess is false and then enabled in EKS console at 11:01, and tags are updated from Rancher before 11:05, then the value is likely to be overwritten. This can also occur if tags are updated while the cluster is still processing the update. The issue described in this example shouldn't occur if the cluster is registered and the PrivateAccess fields are nil. diff --git a/versioned_docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md b/versioned_docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md index 53e7a4c6d9c..8ee65f0c248 100644 --- a/versioned_docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md +++ b/versioned_docs/version-2.7/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md @@ -40,6 +40,10 @@ The AKSConfig, EKSConfig or GKEConfig represents the desired state. Nil values a UpstreamSpec represents the cluster as it is in the hosted Kubernetes provider. It's refreshed every 5 minutes. After the UpstreamSpec is refreshed, Rancher checks if the cluster has an update in progress. If it's currently updating, nothing further is done. If it is not currently updating, any managed fields on AKSConfig, EKSConfig or GKEConfig are overwritten with their corresponding value from the recently updated UpstreamSpec. +:::warning +When you import a cluster from a cloud provider into Rancher, UpstreamSpec represents the cluster state and Config is empty. If you then update the imported cluster through the Rancher UI, both UpstreamSpec and Config become non-null. Any further updates to the cluster should be applied through Rancher. This is because there is no safe way to determine if changes originating from UpstreamSpec represent the desired state or just a mismatch with Config. If you update the imported cluster through the cloud provider console after you apply any updates through the Rancher UI, the controller will deploy a rollback and the content of Config will be considered the desired state. +::: + The effective desired state can be thought of as the UpstreamSpec, plus all non-nil fields in the AKSConfig, EKSConfig or GKEConfig. This is what is displayed in the UI. If Rancher and another source attempt to update a cluster at the same time, or within 5 minutes of an update finishing, any managed fields are likely to get caught in a race condition. To use EKS as an example, a cluster may have PrivateAccess as a managed field. If PrivateAccess is false and then enabled in EKS console at 11:01, and tags are updated from Rancher before 11:05, then the value is likely to be overwritten. This can also occur if tags are updated while the cluster is still processing the update. The issue described in this example shouldn't occur if the cluster is registered and the PrivateAccess fields are nil. diff --git a/versioned_docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md b/versioned_docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md index 53e7a4c6d9c..8ee65f0c248 100644 --- a/versioned_docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md +++ b/versioned_docs/version-2.8/reference-guides/cluster-configuration/rancher-server-configuration/sync-clusters.md @@ -40,6 +40,10 @@ The AKSConfig, EKSConfig or GKEConfig represents the desired state. Nil values a UpstreamSpec represents the cluster as it is in the hosted Kubernetes provider. It's refreshed every 5 minutes. After the UpstreamSpec is refreshed, Rancher checks if the cluster has an update in progress. If it's currently updating, nothing further is done. If it is not currently updating, any managed fields on AKSConfig, EKSConfig or GKEConfig are overwritten with their corresponding value from the recently updated UpstreamSpec. +:::warning +When you import a cluster from a cloud provider into Rancher, UpstreamSpec represents the cluster state and Config is empty. If you then update the imported cluster through the Rancher UI, both UpstreamSpec and Config become non-null. Any further updates to the cluster should be applied through Rancher. This is because there is no safe way to determine if changes originating from UpstreamSpec represent the desired state or just a mismatch with Config. If you update the imported cluster through the cloud provider console after you apply any updates through the Rancher UI, the controller will deploy a rollback and the content of Config will be considered the desired state. +::: + The effective desired state can be thought of as the UpstreamSpec, plus all non-nil fields in the AKSConfig, EKSConfig or GKEConfig. This is what is displayed in the UI. If Rancher and another source attempt to update a cluster at the same time, or within 5 minutes of an update finishing, any managed fields are likely to get caught in a race condition. To use EKS as an example, a cluster may have PrivateAccess as a managed field. If PrivateAccess is false and then enabled in EKS console at 11:01, and tags are updated from Rancher before 11:05, then the value is likely to be overwritten. This can also occur if tags are updated while the cluster is still processing the update. The issue described in this example shouldn't occur if the cluster is registered and the PrivateAccess fields are nil.